optimizing code / DB for a 50,000 row table - php

Question

I have a list of 300 RSS feeds of news articles stored in a database and every few minutes I grab the contents of every single feed. Each feed contains around 10 articles and I want to store each article in a DB.
The Problem: My DB table is over 50,000 rows and rapidly growing; each time I run my script to get new feeds, it's adding at least 100 more rows. It's to the point where my DB is hitting 100% CPU Utilzation.
The Question: How do I optimize my code / DB?
Note: I do not care about my server's CPU (which is <15% when running this). I greatly care about my DB's CPU.
Possible solutions I'm seeing:
Currently, every time the script runs, it goes to $this->set_content_source_cache where it returns an array of array('link', 'link', 'link', etc.) from all the rows in the table. This is used to later cross-reference to make sure there are no duplicating links. Would not doing this and simply changing the DB so the link column is unique speed things up? Possibly throw this array in memcached instead so it has to only create this array once an hour / day?
break statement if the link is set so that it moves on to the next source?
only checking links that are less than a week old?
Here's what I'm doing:
//$this->set_content_source_cache goes through all 50,000 rows and adds each link to an array so that it's array('link', 'link', 'link', etc.)
$cache_source_array = $this->set_content_source_cache();
$qry = "select source, source_id, source_name, geography_id, industry_id from content_source";
foreach($this->sql->result($qry) as $row_source) {
$feed = simplexml_load_file($row_source['source']);
if(!empty($feed)) {
for ($i=0; $i < 10 ; $i++) {
// most often there are only 10 feeds per rss. Since we check every 2 minutes, if there are
// a few more, then meh, we probably got it last time around
if(!empty($feed->channel->item[$i])) {
// make sure that the item is not blank
$title = $feed->channel->item[$i]->title;
$content = $feed->channel->item[$i]->description;
$link = $feed->channel->item[$i]->link;
$pubdate = $feed->channel->item[$i]->pubdate;
$source_id = $row_source['source_id'];
$source_name = $row_source['source_name'];
$geography_id = $row_source['geography_id'];
$industry_id = $row_source['industry_id'];
// random stuff in here to each link / article to make it data-worthy
if(!isset($cache_source_array[$link])) {
// start the transaction
$this->db->trans_start();
$qry = "insert into content (headline, content, link, article_date, status, source_id, source_name, ".
"industry_id, geography_id) VALUES ".
"(?, ?, ?, ?, 2, ?, ?, ?, ?)";
$this->db->query($qry, array($title, $content, $link, $pubdate, $source_id, $source_name, $industry_id, $geography_id));
// this is my framework's version of mysqli_insert_id()
$content_id = $this->db->insert_id();
$qry = "insert into content_ratings (content_id, comment_count, company_count, contact_count, report_count, read_count) VALUES ".
"($content_id, '0', '0', 0, '0', '0')";
$result2 = $this->db->query($qry);
$this->db->trans_complete();
if($this->db->trans_status() == TRUE) {
$cache_source_array[$link] = $content_id;
echo "Good!<br />";
} else {
echo "Bad!<br />";
}
} else {
// link alread exists
echo "link exists!";
}
}
}
} else {
// feed is empty
}
}
}

Answer 1

I think you answered your own question:
Currently, every time the script runs, it goes to
$this->set_content_source_cache where it returns an array of
array('link', 'link', 'link', etc.) from all the rows in the table.
This is used to later cross-reference to make sure there are no
duplicating links. Would not doing this and simply changing the DB so
the link column is unique speed things up?
Yes, creating a primary key or unique index and allowing the DB to throw an error if there is a duplicate is a much better practice and should be much more efficient.
REFERENCE EDIT:
mysql 5.0 indexes - Unique vs Non Unique
http://dev.mysql.com/doc/refman/5.0/en/create-index.html