CURL server data transfer timeout - php

Question

I am using the following code to get the xml data form icecat.biz:
set_time_limit (0);
$login = "Arpan";
$password = "arpan";
//$url="http://data.icecat.biz/export/freexml.int/EN/files.index.xml";
$url= "http://data.icecat.biz/export/level4/EN";
//$url="http://data.icecat.biz/export/freexml.int/DE/10.xml";
$user_agent = 'Mozilla/5.0 (Windows; U;
Windows NT 5.1; ru; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9';
$header = array(
"Accept: text/xml,application/xml,application/xhtml+xml,
text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
"Accept-Language: ru-ru,ru;q=0.7,en-us;q=0.5,en;q=0.3",
"Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7",
"Keep-Alive: 300");
$local_path = "myxml.xml";
$file_handle = fopen($local_path, "w");
ob_start();
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FILE, $file_handle);
curl_setopt($ch, CURLOPT_HEADER, 0);
//curl_setopt ( $ch , CURLOPT_HTTPHEADER, $header );
curl_setopt($ch, CURLOPT_USERPWD, $login . ":" . $password);
curl_setopt($ch, CURLOPT_TIMEOUT, 0); // times out after 4s
//curl_setopt($c, CURLOPT_TIMEOUT, 2);
//curl_setopt($ch, CURLOPT_NOBODY, TRUE); // remove body
//curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
//$head = curl_exec($ch);
$result = curl_exec ($ch);
if(curl_errno($ch))
{
echo 'Curl error: ' . curl_error($ch);
}
curl_close($ch);
ob_end_clean();
fclose ($file_handle);
$xmlStr = file_get_contents($local_path);
$xmlObj = simplexml_load_string($xmlStr);
print "<pre>";
//print_r($xmlObj->Product->ProductRelated->attributes()->ID);
print_r($xmlObj);
exit;
The page is being executed for a unlimited time but the XML is not being updated after 10 to 20 sec. The output xml is also not being completed. I think after a certain time the server is not responding or data is not being transferred.
Here is the error message:
**** The server xml (icecat) size is big
What is the problem and how do I fix it?

Answer 1

Sounds like you are not giving enough time for the request to download properly.
Uncomment your //curl_setopt($c, CURLOPT_TIMEOUT, 2);
And put the timeout to 600 for a test.
Beyond that your request looks fine, you could always check to see if the server is caching responses, the last thing that I've seen recently happening is if some of my users have reverse proxies in order to cache their normal operations. Where some truncated responses got cached and thats all they got back for a 24 hour period although that may not be related to you.