cURL not processing all requests - stops around 110 - php

Question

So I am having trouble getting this code to work. I pulled it from a blog and it is based on the wordpress link checker. I have about 6000 urls in a database i need to check the http status so this seems like a great choice. I've modified the code slightly to fit my needs and it works (kind of).
I have checked the url_list array throughout the code and it contains all of the url's. The problem is that it will basically stop executing after about the 110th row, it is kind of random but generally around that number. Not really sure if I need to set a timeout somewhere or if I have a bug in the code. I noticed that if I set $max_connections greater than 8 it will return a 500 error. Any suggestions?
<?php
// CONFIG
$db_host = 'localhost';
$db_user = 'test';
$db_pass = 'yearight';
$db_name = 'URLS';
$excluded_domains = array();
$max_connections = 7;
$dbh = new PDO('mysql:host=localhost;dbname=URLS', $db_user, $db_pass);
$sth = $dbh->prepare("SELECT url FROM list");
$sth->execute();
$result = $sth->fetchAll(PDO::FETCH_COLUMN, 0);
// initialize some variables
$url_list = array();
$working_urls = array();
$dead_urls = array();
$not_found_urls = array();
$active = null;
foreach($result as $d) {
// get all links via regex
if (preg_match_all('#((http?://)?([-\w]+\.[-\w\.]+)+\w(:\d+)?(/([-\w/_\.]*(\?\S+)?)?)*)#', $d, $matches)) {
foreach ($matches[1] as $url) {
// store the url
$url_list []= $url;
}
}
}
// 1. multi handle
$mh = curl_multi_init();
// 2. add multiple URLs to the multi handle
for ($i = 0; $i < $max_connections; $i++) {
add_url_to_multi_handle($mh, $url_list);
}
// 3. initial execution
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
// 4. main loop
while ($active && $mrc == CURLM_OK) {
// 5. there is activity
if (curl_multi_select($mh) != -1) {
// 6. do work
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
// 7. is there info?
if ($mhinfo = curl_multi_info_read($mh)) {
// this means one of the requests were finished
// 8. get the info on the curl handle
$chinfo = curl_getinfo($mhinfo['handle']);
// 9. dead link?
if (!$chinfo['http_code']) {
$dead_urls []= $chinfo['url'];
// 10. 404?
} else if ($chinfo['http_code'] == 404) {
$not_found_urls []= $chinfo['url'];
// 11. working
} else {
$working_urls []= $chinfo['url'];
}
// 12. remove the handle
curl_multi_remove_handle($mh, $mhinfo['handle']);
curl_close($mhinfo['handle']);
// 13. add a new url and do work
if (add_url_to_multi_handle($mh, $url_list)) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
}
}
// 14. finished
curl_multi_close($mh);
echo "==Dead URLs==<br/>";
echo implode("<br/>",$dead_urls) . "<br/><br/>";
echo "==404 URLs==<br>";
echo implode("<br/>",$not_found_urls) . "<br/><br/>";
echo "==Working URLs==<br/>";
echo implode("<br/>",$working_urls);
echo "<pre>";
var_dump($url_list);
echo "</pre>";
// 15. adds a url to the multi handle
function add_url_to_multi_handle($mh, $url_list) {
static $index = 0;
// if we have another url to get
if ($url_list[$index]) {
// new curl handle
$ch = curl_init();
// set the url
curl_setopt($ch, CURLOPT_URL, $url_list[$index]);
// to prevent the response from being outputted
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// follow redirections
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
// do not need the body. this saves bandwidth and time
curl_setopt($ch, CURLOPT_NOBODY, 1);
// add it to the multi handle
curl_multi_add_handle($mh, $ch);
// increment so next url is used next time
$index++;
return true;
} else {
// we are done adding new URLs
return false;
}
}
?>
UPDATE:
I have written a script in bash that does the same thing as this. I notice when I was going through the text file the info was output to, that when it fails it is typically around links that return odd http status codes like 000 and 522 some of them tend to execute for up to 5 minutes! So I am wondering if the PHP version of cURL is stopping execution when it encounters these status codes. It is just a thought and might add more value to help solve the issue.

Answer 1

1 - EXECUTION TIME ISSUE
2 - DECLAE AT THE TOP OF CODE MAX_EXECUTION_TIME, WILL HELP THIS FOR SURE
bool set_time_limit ( int $seconds )