Does file_get_contents() have a timeout setting? - php

I am calling a series of links using the file_get_contents() method in a loop. Each link may take more than 15 minutes to process. Now, I worry about whether PHP's file_get_contents() has a timeout period?
If yes, it will time out with a call and move to next link. I don't want to call the next link without the prior one finishing.
So, please tell me whether file_get_contents() has a timeout period. The file which contains the file_get_contents() is set to set_time_limit() to zero (unlimited).

The default timeout is defined by default_socket_timeout ini-setting, which is 60 seconds. You can also change it on the fly:
ini_set('default_socket_timeout', 900); // 900 Seconds = 15 Minutes
Another way to set a timeout, would be to use stream_context_create to set the timeout as HTTP context options of the HTTP stream wrapper in use:
$ctx = stream_context_create(array('http'=>
array(
'timeout' => 1200, //1200 Seconds is 20 Minutes
)
));
echo file_get_contents('http://example.com/', false, $ctx);

As #diyism mentioned, "default_socket_timeout, stream_set_timeout, and stream_context_create timeout are all the timeout of every line read/write, not the whole connection timeout." And the top answer by #stewe has failed me.
As an alternative to using file_get_contents, you can always use curl with a timeout.
So here's a working code that works for calling links.
$url='http://example.com/';
$ch=curl_init();
$timeout=5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$result=curl_exec($ch);
curl_close($ch);
echo $result;

Yes! By passing a stream context in the third parameter:
Here with a timeout of 1s:
file_get_contents("https://abcedef.com", 0, stream_context_create(["http"=>["timeout"=>1]]));
Source in comment section of https://www.php.net/manual/en/function.file-get-contents.php
HTTP context options:
method
header
user_agent
content
request_fulluri
follow_location
max_redirects
protocol_version
timeout
Non HTTP stream contexts
Socket
FTP
SSL
CURL
Phar
Context (notifications callback)
Zip

It is worth noting that if changing default_socket_timeout on the fly, it might be useful to restore its value after your file_get_contents call:
$default_socket_timeout = ini_get('default_socket_timeout');
....
ini_set('default_socket_timeout', 10);
file_get_contents($url);
...
ini_set('default_socket_timeout', $default_socket_timeout);

For me work when i change my php.ini in my host:
; Default timeout for socket based streams (seconds)
default_socket_timeout = 300

For prototyping, using curl from the shell with the -m parameter allow to pass milliseconds, and will work in both cases, either the connection didn't initiate, error 404, 500, bad url, or the whole data wasn't retrieved in full in the allowed time range, the timeout is always effective. Php won't ever hang out.
Simply don't pass unsanitized user data in the shell call.
system("curl -m 50 -X GET 'https://api.kraken.com/0/public/OHLC?pair=LTCUSDT&interval=60' -H 'accept: application/json' > data.json");
// This data had been refreshed in less than 50ms
var_dump(json_decode(file_get_contents("data.json"),true));

Related

why curl timeout when get from url?

why would cURL in PHP return timeout message when get HTML from web page?
Here is the PHP code.
function getFromUrl( $url )
{
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($curl);
if (curl_errno($curl))
{
echo 'Error:' . curl_error($curl) . '<br>' ;
}
curl_close($curl);
return $result ;
}
I get the expected results when I run the function with www.google.com as the URL.
$url = 'http://www.google.com' ;
$result = getFromUrl($url) ;
But, when I pass in the URL of web page on a 2nd web server, I get a timeout response. The URL exists when I paste it into a browser. Why the timeout message?
$url = "http://xxx.54.20.170:10080/accounting/tester/hello.html" ;
echo $url . '<br>' ;
$rv = getFromUrl( $url ) ;
echo $rv . '<br>' ;
here is the cURL error message:
Error:Failed to connect to xxx.54.20.170 port 10080: Connection timed out
I am looking to transfer data from one web server to another.
thanks,
For PHP,
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 400); //timeout in seconds
From terminal first check if curl is working using below extra options.
--connect-timeout
Maximum time in seconds that you allow the connection to the
server to take. This only limits the connection phase, once
curl has connected this option is of no more use. Since 7.32.0,
this option accepts decimal values, but the actual timeout will
decrease in accuracy as the specified timeout increases in deci‐
mal precision. See also the -m, --max-time option.
If this option is used several times, the last one will be used.
and
-m, --max-time
Maximum time in seconds that you allow the whole operation to
take. This is useful for preventing your batch jobs from hang‐
ing for hours due to slow networks or links going down. Since
7.32.0, this option accepts decimal values, but the actual time‐
out will decrease in accuracy as the specified timeout increases
in decimal precision. See also the --connect-timeout option.
If this option is used several times, the last one will be used.
Try to use them to increase timeout time.
There are many reasons for curl not working. Some of them can be,
1) Response time is slow.
2) Few site has check on few header parameters to respond to request. These parameters include User-Agent, Referer, etc to make sure it is coming from valid source and not through bots.

Page executes very slow while curl runs

here i explain details about my question
first check below code
$ch = curl_init('http://example123.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $xml);
$result = #curl_exec($ch);
now my question is that if "http://example123.com" is not valid or there is no such URL, then what is the problem.
i have a page there written the above. while execute the code the page takes too much time. but if i comment above 5 line then my page executes faster.
can anybody told me what is the reason behind and why the page execute very slow.
Thanks Sanjib
Your script waits for a response (which may take 60 seconds for default_socket_timeout.)
You should set curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); to make the script redirect from http://example123.com/ to http://ww38.example123.com/, the way it does in browser.
When cUrl request encounters INVALID URI it waits until Default connection timeout is reached, this makes page load slower.
Default Connection timeout set in lib/connect.h (if Linux Server)
You can change it here
#define DEFAULT_CONNECT_TIMEOUT 300000 /* milliseconds == five minutes */
Or you can Explicitly set this in you Codes
curl_setopt( $c, CURLOPT_CONNECTIONTIMEOUT, 100 ); # Or curl_setopt($ch, CURLOPT_CONNECTTIMEOUT_MS ,0);
curl_setopt($ch, CURLOPT_TIMEOUT, 400); # timeout in seconds
CURLOPT_CONNECTTIMEOUT : The number of seconds to wait while trying to connect. Use 0 to wait indefinitely.
CURLOPT_TIMEOUT : The maximum number of seconds to allow cURL functions to execute.
*** (If you are using PHP as a fastCGI application then make sure you check the fastCGI timeout settings.)*

PHP file_get_contents very slow when using full url

I am working with a script (that I did not create originally) that generates a pdf file from an HTML page. The problem is that it is now taking a very long time, like 1-2 minutes, to process. Supposedly this was working fine originally, but has slowed down within the past couple of weeks.
The script calls file_get_contents on a php script, which then outputs the result into an HTML file on the server, and runs the pdf generator app on that file.
I seem to have narrowed down the problem to the file_get_contents call on a full url, rather than a local path.
When I use
$content = file_get_contents('test.txt');
it processes almost instantaneously. However, if I use the full url
$content = file_get_contents('http://example.com/test.txt');
it takes anywhere from 30-90 seconds to process.
It's not limited to our server, it is slow when accessing any external url, such as http://www.google.com. I believe the script calls the full url because there are query string variables that are necessary that don't work if you call the file locally.
I also tried fopen, readfile, and curl, and they were all similarly slow. Any ideas on where to look to fix this?
Note: This has been fixed in PHP 5.6.14. A Connection: close header will now automatically be sent even for HTTP/1.0 requests. See commit 4b1dff6.
I had a hard time figuring out the cause of the slowness of file_get_contents scripts.
By analyzing it with Wireshark, the issue (in my case and probably yours too) was that the remote web server DIDN'T CLOSE THE TCP CONNECTION UNTIL 15 SECONDS (i.e. "keep-alive").
Indeed, file_get_contents doesn't send a "connection" HTTP header, so the remote web server considers by default that's it's a keep-alive connection and doesn't close the TCP stream until 15 seconds (It might not be a standard value - depends on the server conf).
A normal browser would consider the page is fully loaded if the HTTP payload length reaches the length specified in the response Content-Length HTTP header. File_get_contents doesn't do this and that's a shame.
SOLUTION
SO, if you want to know the solution, here it is:
$context = stream_context_create(array('http' => array('header'=>'Connection: close\r\n')));
file_get_contents("http://www.something.com/somepage.html",false,$context);
The thing is just to tell the remote web server to close the connection when the download is complete, as file_get_contents isn't intelligent enough to do it by itself using the response Content-Length HTTP header.
I would use curl() to fetch external content, as this is much quicker than the file_get_contents method. Not sure if this will solve the issue, but worth a shot.
Also note that your servers speed will effect the time it takes to retrieve the file.
Here is an example of usage:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://example.com/test.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
Sometimes, it's because the DNS is too slow on your server, try this:
replace
echo file_get_contents('http://www.google.com');
as
$context=stream_context_create(array('http' => array('header'=>"Host: www.google.com\r\n")));
echo file_get_contents('http://74.125.71.103', false, $context);
I had the same issue,
The only thing that worked for me is setting timeout in $options array.
$options = array(
'http' => array(
'header' => implode($headers, "\r\n"),
'method' => 'POST',
'content' => '',
'timeout' => .5
),
);
$context = stream_context_create(array('http' => array('header'=>'Connection: close\r\n')));
$string = file_get_contents("http://localhost/testcall/request.php",false,$context);
Time: 50976 ms (avaerage time in total 5 attempts)
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, "http://localhost/testcall/request.php");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
echo $data = curl_exec($ch);
curl_close($ch);
Time: 46679 ms (avaerage time in total 5 attempts)
Note: request.php is used to fetch some data from mysql database.
Can you try fetching that url, on the server, from the command line? curl or wget come to mind. If those retrieve the URL at a normal speed, then it's not a network problem and most likely something in the apache/php setup.
I have a huge data passed by API, I'm using file_get_contents to read the data, but it took around 60 seconds. However, using KrisWebDev's solution it took around 25 seconds.
$context = stream_context_create(array('https' => array('header'=>'Connection: close\r\n')));
file_get_contents($url,false,$context);
What I would also consider with Curl is that you can "thread" the requests. This has helped me immensely as I do not have access to a version of PHP that allows threading at the moment .
For example, I was getting 7 images from a remote server using file_get_contents and it was taking 2-5 seconds per request. This process alone was adding 30seconds or something to the process, while the user waited for the PDF to be generated.
This literally reduced the time to about 1 image. Another example, I verify 36 urls in the time it took before to do one. I think you get the point. :-)
$timeout = 30;
$retTxfr = 1;
$user = '';
$pass = '';
$master = curl_multi_init();
$node_count = count($curlList);
$keys = array("url");
for ($i = 0; $i < $node_count; $i++) {
foreach ($keys as $key) {
if (empty($curlList[$i][$key])) continue;
$ch[$i][$key] = curl_init($curlList[$i][$key]);
curl_setopt($ch[$i][$key], CURLOPT_TIMEOUT, $timeout); // -- timeout after X seconds
curl_setopt($ch[$i][$key], CURLOPT_RETURNTRANSFER, $retTxfr);
curl_setopt($ch[$i][$key], CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch[$i][$key], CURLOPT_USERPWD, "{$user}:{$pass}");
curl_setopt($ch[$i][$key], CURLOPT_RETURNTRANSFER, true);
curl_multi_add_handle($master, $ch[$i][$key]);
}
}
// -- get all requests at once, finish when done or timeout met --
do { curl_multi_exec($master, $running); }
while ($running > 0);
Then check over the results:
if ((int)curl_getinfo($ch[$i][$key], CURLINFO_HTTP_CODE) > 399 || empty($results[$i][$key])) {
unset($results[$i][$key]);
} else {
$results[$i]["options"] = $curlList[$i]["options"];
}
curl_multi_remove_handle($master, $ch[$i][$key]);
curl_close($ch[$i][$key]);
then close file:
curl_multi_close($master);
I know that is old question but I found it today and answers didn't work for me. I didn't see anyone saying that max connections per IP may be set to 1. That way you are doing API request and API is doing another request because you use full url. That's why loading directly from disc works. For me that fixed a problem:
if (strpos($file->url, env('APP_URL')) === 0) {
$url = substr($file->url, strlen(env('APP_URL')));
} else {
$url = $file->url;
}
return file_get_contents($url);

Curl timeout issue

Using PHP's Curl how do I know that the page is not responding so as to grab another one?
Use the CURLOPT_CONNECTTIMEOUT option:
// Wait two seconds before bailing
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 2);
There's also CURLOPT_TIMEOUT, which works for an entire request call (including DNS fetching and reading the data).
To check if a call did time out, you may be able to check its return value. If not, the CURL handler's curl_errno is set, which you can compare to CURLE_OPERATION_TIMEDOUT (or just CURLE_OK).

Setting Curl's Timeout in PHP

I'm running a curl request on an eXist database through php. The dataset is very large, and as a result, the database consistently takes a long amount of time to return an XML response. To fix that, we set up a curl request, with what is supposed to be a long timeout.
$ch = curl_init();
$headers["Content-Length"] = strlen($postString);
$headers["User-Agent"] = "Curl/1.0";
curl_setopt($ch, CURLOPT_URL, $requestUrl);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERPWD, 'admin:');
curl_setopt($ch,CURLOPT_TIMEOUT,1000);
$response = curl_exec($ch);
curl_close($ch);
However, the curl request consistently ends before the request is completed (<1000 when requested via a browser). Does anyone know if this is the proper way to set timeouts in curl?
See documentation: http://www.php.net/manual/en/function.curl-setopt.php
CURLOPT_CONNECTTIMEOUT - The number of seconds to wait while trying to connect. Use 0 to wait indefinitely.
CURLOPT_TIMEOUT - The maximum number of seconds to allow cURL functions to execute.
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 400); //timeout in seconds
also don't forget to enlarge time execution of php script self:
set_time_limit(0);// to infinity for example
Hmm, it looks to me like CURLOPT_TIMEOUT defines the amount of time that any cURL function is allowed to take to execute. I think you should actually be looking at CURLOPT_CONNECTTIMEOUT instead, since that tells cURL the maximum amount of time to wait for the connection to complete.
There is a quirk with this that might be relevant for some people... From the PHP docs comments.
If you want cURL to timeout in less than one second, you can use CURLOPT_TIMEOUT_MS, although there is a bug/"feature" on "Unix-like systems" that causes libcurl to timeout immediately if the value is < 1000 ms with the error "cURL Error (28): Timeout was reached". The explanation for this behavior is:
"If libcurl is built to use the standard system name resolver, that portion of the transfer will still use full-second resolution for timeouts with a minimum timeout allowed of one second."
What this means to PHP developers is "You can't use this function without testing it first, because you can't tell if libcurl is using the standard system name resolver (but you can be pretty sure it is)"
The problem is that on (Li|U)nix, when libcurl uses the standard name resolver, a SIGALRM is raised during name resolution which libcurl thinks is the timeout alarm.
The solution is to disable signals using CURLOPT_NOSIGNAL. Here's an example script that requests itself causing a 10-second delay so you can test timeouts:
if (!isset($_GET['foo'])) {
// Client
$ch = curl_init('http://localhost/test/test_timeout.php?foo=bar');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_NOSIGNAL, 1);
curl_setopt($ch, CURLOPT_TIMEOUT_MS, 200);
$data = curl_exec($ch);
$curl_errno = curl_errno($ch);
$curl_error = curl_error($ch);
curl_close($ch);
if ($curl_errno > 0) {
echo "cURL Error ($curl_errno): $curl_error\n";
} else {
echo "Data received: $data\n";
}
} else {
// Server
sleep(10);
echo "Done.";
}
From http://www.php.net/manual/en/function.curl-setopt.php#104597
Your code sets the timeout to 1000 seconds. For milliseconds, use CURLOPT_TIMEOUT_MS.
You will need to make sure about timeouts between you and the file.
In this case PHP and Curl.
To tell Curl to never timeout when a transfer is still active, you need to set CURLOPT_TIMEOUT to 0, instead of 1000.
curl_setopt($ch, CURLOPT_TIMEOUT, 0);
In PHP, again, you must remove time limits or PHP it self (after 30 seconds by default) will kill the script along Curl's request. This alone should fix your issue.
In addition, if you require data integrity, you could add a layer of security by using ignore_user_abort:
# The maximum execution time, in seconds. If set to zero, no time limit is imposed.
set_time_limit(0);
# Make sure to keep alive the script when a client disconnect.
ignore_user_abort(true);
A client disconnection will interrupt the execution of the script and possibly damaging data,
eg. non-transitional database query, building a config file, ecc., while in your case it would download a partial file... and you might, or not, care about this.
Answering this old question because this thread is at the top on engine searches for CURL_TIMEOUT.
You can't run the request from a browser, it will timeout waiting for the server running the CURL request to respond. The browser is probably timing out in 1-2 minutes, the default network timeout.
You need to run it from the command line/terminal.
If you are using PHP as a fastCGI application then make sure you check the fastCGI timeout settings.
See: PHP curl put 500 error

Categories