php url check available - php

Question

I want to check whether url is available from my database. I choose fopen, but I test 30 rows from my database, it will cost nearly 20 seconds. Is there any way, can make it more efficient? Thanks.
<?php
$start_t = microtime(true);
//connect database and select query
while ($row = mysql_fetch_array($result)){
//$url = 'http://www.google.com'; //not test from database, but a google.com, one url will cost 0.49 seconds.
$url = $row['url'];
$res = #fopen($url, "r ");
if($res){
echo $row['url'].' yes<br />';
}else{
echo $row['url']. ' no<br />';
}
}
$end_t = microtime(true);
$totaltime = $end_t-$start_t;
echo "<br />".$totaltime." s";
?>

Answer 1

Try using fsockopen which is faster than fopen
<?php
$t = microtime(true);
$valid = #fsockopen("www.google.com", 80, $errno, $errstr, 30);
echo (microtime(true)-$t);
if (!$valid) {
echo "Failure";
} else {
echo "Success";
}
?>
Output:
0.0013298988342285

Answer 2

You can try using CURL with the CURLOPT_NOBODY option set, which uses the HTTP HEAD method and avoids downloading the entire page:
$ch = curl_init($row['url']);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_exec($ch);
$retcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
// 400 means not found, 200 means found.
curl_close($ch);
From the CURLOPT_NOBODY documentation:
TRUE to exclude the body from the
output. Request method is then set to
HEAD. Changing this to FALSE does not
change it to GET.

Answer 3

Try Bulk URL check, that is, in blocks of 10 or 20
Curl Multi Exec.
http://semlabs.co.uk/journal/object-oriented-curl-class-with-multi-threading
Use the CURL options for NOBODY and HEADER ONLY, so your response will be much faster.
Also dont forget to put TIMEOUT for curl, else one BAD url may take too much time.
i was doing 50 URL checks in 20 secs.
Hope that Helps.

Answer 4

You can't speed things up like that.
With 30 rows I assume you are connecting to 30 different urls. 20 seconds is already a good time for that.
Also I suggest you to use file_get_contents to retrive HTML
Or if you need to know the header response use get_headers();
If you want to speed up the process just spawn more process. Each of them will fetch a tot urls.
Addendum
Also don't forget about the great Zend_HTTP_Client(); that is very good for such task