PHP Website Crawler Data Extraction Multiple Loop Error 404 - php

Question

I am looking to crawl multiple gig listing websites to compile a ultimate listing guide with links back to the original websites.
A lot of these websites don't have an API so I have to use a rather crudely put together php script that will extract the data I require. (eg date, venue, country etc)
Most sites have a fairly easy to use directory of gigs, but on certain sites, they require manually inputting information to get "relevant" shows to you.
So to get around this, I created a loop that worked on the basis of:
page.php?id=$counter+1
So it finds the last inserted gig into the db and carries on getting data for the next 100 or so.
BUT this only works on the condition that the gigs on the site will continue numerically accurately, and of course, they don't due to cancellations etc.
This leaves me with the wonderful
Warning: file_get_contents(http://www.domain.com/show/page.php?id=123456) [function.file-get-contents]: failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in...
How is it possible to create a loop that will be able to skip these errors and carry on rather than just sitting on them?
Below is the entire code (Limit of +5 at the moment for testing)
include_once('simple_html_dom.php');
$cntqry = mysql_query("SELECT * FROM `gigs_tbl` ORDER BY `counter` DESC LIMIT 1");
$cntnum = mysql_num_rows($cntqry);
if($cntnum!=0)
{
$cntget = mysql_fetch_assoc($cntqry);
$start = $cntget['counter'];
}
else {
$start = 10767799;
}
$counter = 0;
$limit = $start +5;
for($start; $start < $limit; $start++) {
$counter = $start + 1;
$target_url = "http://www.domain.com/show/page.php?id=$counter";
$html = new simple_html_dom();
$html->load_file($target_url);
foreach($html->find('div[class=vevent]') as $showrow){
$artist = strip_tags($showrow->find('h2',0));
$genre = strip_tags($html->find('span[class=genre]',0));
$venue = strip_tags($showrow->find('span[class=location]',0));
$street = strip_tags($html->find('span[itemprop=streetAddress]',0));
$locality = strip_tags($html->find('span[itemprop=addressLocality]',0));
$postcode = strip_tags($html->find('span[itemprop=postalCode]',0));
$country = strip_tags($html->find('span[itemprop=addressRegion]',0));
$originalDate = strip_tags($html->find('meta[itemprop=startDate]',0)->content);
$newDate = date("U", strtotime($originalDate));
// INSERT
mysql_query("INSERT INTO `gigs_tbl` VALUES('','$counter','$newDate','$venue','$street','$locality','$postcode','$country','$gen re','$artist','reverbnation')");
}
}
Ten virtual high fives to anyone who can guess which website is causing this issue ;)

Answer 1

find() returns NULL if nothing found... So, a way to do what you want is to exploit this :)
Since your didnt provide the real link, here's an example explaining how:
$start = 'u';
for($start; $start < 'x'; $start++) {
// The only correct url is => http://sourceforge.net/p/mingw/bugs/
$target_url = "http://sourceforge.net/p/ming".$start."/bugs/";
echo "<br/> Link: $target_url";
// #: supresses the errors when the page doesnt exist
$data_string = #file_get_contents($target_url);
$html = new simple_html_dom();
// Load HTML from a string
$html->load($data_string);
// Find returns NULL if nothing found
$search_elements = $html->find('#nav_menu_holder h1');
if($search_elements) {
echo "<br/> Page FOUND. Title => " . $search_elements[0]->plaintext;
}
else {
echo "<br/> Page NOT FOUND !!";
}
echo "<hr>";
// Clear DOM object
$html->clear();
unset($html);
}
PHP-Fiddle DEMO