I have a simple PHP script that uses cURL to grab the contents of a URL:
$curlOptions = [
CURLOPT_URL => $url,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HEADER => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_MAXREDIRS => 5,
CURLOPT_CONNECTTIMEOUT => 3,
CURLOPT_TIMEOUT => 5,
CURLOPT_NOBODY => TRUE,
CURLOPT_RETURNTRANSFER => TRUE,
];
$curl = curl_init();
curl_setopt_array( $curl, $curlOptions );
$curlResult = curl_exec( $curl );
$status = curl_getinfo( $curl, CURLINFO_HTTP_CODE );
curl_close( $curl );
However, in some circumstances I clearly see durations of 20 seconds or more eventhough I specified lower time out values for CURLOPT_CONNECTTIMEOUT and CURLOPT_TIMEOUT.
How can I make cURL time out at the values I specified?
UPDATE:
All the cURL's that take 20 seconds to complete return a "status" of 0. Its probably a DNS problem where it can't resolve the host. I would assume that CURLOPT_CONNECTTIMEOUT would take care of this?
It seems that your code works perfectly.
I've created a simple sleep page, you can check it (gets sleep time as input).
In case that you find cases where the code did not time out, I suggest checking the status code, and to see whether the code got hung on other part (maybe not the curl itself).
<?php
$url = "http://funnify.me/sleep.php?sleep=12"; // time in seconds
$curlOptions = [
CURLOPT_URL => $url,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HEADER => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_MAXREDIRS => 5,
CURLOPT_CONNECTTIMEOUT => 8,
CURLOPT_TIMEOUT => 8,
CURLOPT_NOBODY => TRUE,
CURLOPT_RETURNTRANSFER => TRUE,
];
$curl = curl_init();
curl_setopt_array( $curl, $curlOptions );
$curlResult = curl_exec( $curl );
$status = curl_getinfo( $curl, CURLINFO_HTTP_CODE );
curl_close( $curl );
echo $status;
?>
Cheers,
Ika
Related
When I'm trying to send data on a API using CURL. API taking long time (10 minute or more) to the execution and at the end API successfully executed on its destination. but After the 10 minute, curl not returning any response.
But same process are complete in under the 5 minutes, then everything is working fine.
Here is my code
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt_array($ch, array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => "postvar1=value1",
CURLOPT_HTTPHEADER => array(
"Content-Type: application/x-www-form-urlencoded",
),
));
$api_res = curl_exec($ch);
$err = curl_error($ch);
curl_close($ch);
$api_obj = json_decode($api_res, true);
Please suggest me the better solution for successfully run this script after 10 minute or more time.
I would first test PHP itself, to see if it can run for 10 minutes or longer. Your question doesn't tell us whether you tested this.
You could use this little script:
<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
$start = time();
while (time() + (10 * 60) > $start) {
1 + 2 + 3;
}
echo "I am still here!";
If your PHP can run for 10 minutes or more, you should see the: "I am still here!", message, otherwise it will post the error: "Fatal error: Maximum execution time of ?? seconds exceeded.".
You cannot use sleep() for this, because that will actually pause execution, and therefore stops the execution time timer.
Tips on how to extend the execution time of PHP can be found everywhere.
If PHP can run for 10 minutes or longer then you now know that the problem could indeed be your CURL code, or the API server.
Use this...
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => 'API_URL',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'POST',
CURLOPT_POSTFIELDS =>'{
"Param1": 'value1',
"Param2": 'value2'
}',
CURLOPT_HTTPHEADER => array(
'Authorization: Auth_KEY_if_you_have',
'Content-Type: application/x-www-form-urlencoded'
),
));
$response = curl_exec($curl);
curl_close($curl);
echo $response;
I have this code:
<?php
$data = array('name' => 'Ross', 'php_master' => true);
$url = 'http://dsaasd.adsds.nl:8081/cops.nsf/orders?openagent';
$html_brand = "www.google.com";
$ch = curl_init();
$options = array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => "",
CURLOPT_AUTOREFERER => true,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
CURLOPT_VERBOSE => true,
);
curl_setopt_array( $ch, $options );
$response = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ( $httpCode != 200 ){
echo "Return code is {$httpCode} \n"
.curl_error($ch);
} else {
echo "<pre>".htmlspecialchars($response)."</pre>";
}
curl_close($ch);
?>
The problem is, if i send curl post to:
$url = 'http://dsaasd.adsds.nl:8081/cops.nsf/orders?openagent';
Then it does NOT work.
But if i send curl post to the same service but on a different server then it works, this is the other server
$url = 'http://dsaasd.adsds.nl:80/cops.nsf/orders?openagent';
I also post data by normal form post to:
$url = 'http://dsaasd.adsds.nl:8081/cops.nsf/orders?openagent';
And then it works and i receive data on the server.
But with this curl post i keep getting:
Return code is 0 Failed to connect to 11.43.45.123: Network is unreachable
Anyone have any idea?
I think you need to add CURLOPT_POST to the $options array. Docs here.
I'm crawling a few websites, everything it's working fine,
but .... I have one specific website that I'm trying to crawl,
and it's making a few "redirects" before landing to the web I want.
So it's something like ...
http://www.example.com/?day=01/01/2016&action=search_prices
this will go to http://www.example.com/search/default.aspx take a few seconds to search the answer page and then show it on there.
Is there any way to easily do this? any hint, clue, etc would be awesome
Simple code right now (almost all the sites I was crawling were jsons):
function get_web_page( $url ){
$options = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_USERAGENT => "spider", // who am i
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_HTTPHEADER => array('HeaderName: HeaderValue'),
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
CURLOPT_SSL_VERIFYPEER => false // Disabled SSL Cert checks
);
$ch = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$header['errno'] = $err;
$header['errmsg'] = $errmsg;
$header['content'] = $content;
return $header;
}
I am trying to convert this VB script to PHP curl
xmlServerHttp.open "POST","url",False xmlServerHttp.setRequestHeader "Content-Type","application/x-www-form-urlencoded"
xmlServerHttp.send "xmlmessage=" & Server.URLEncode(xmlDocument)
‘ xmlDocument = the Xml Document contain the actual request
xmlServerStatus = xmlServerHttp.status
if xmlServerStatus = "200" then
xmlServerResponse = xmlServerHttp.responseText
Else
Response.Appendtolog ".xmlServer status is " & xmlServerStatus
end if
This is what I have so far however it is failing
$curl = curl_init(url);
// Configuring curl options
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HTTPHEADER => array('Content-type: application/x-www-form-urlencoded') ,
CURLOPT_POSTFIELDS => $xmldoc,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_SSL_VERIFYPEER => false
);
// Setting curl options
curl_setopt_array( $curl, $options );
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, 'POST');
// Getting results
echo curl_exec($curl);
The API i am calling return that the xmlmessage variable is not a valid xml document.
try to change $xmlDoc to $xmlDoc->asXML()
like this
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HTTPHEADER => array('Content-type: application/x-www-form-urlencoded') ,
CURLOPT_POSTFIELDS => array('xmlmessage='=> $xmlDoc),
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_SSL_VERIFYPEER => false
);
I had the below code working to get a HTTP url follow the redirect and then pass back the new page url it was on.
// Follow URL
private function follow_url($url) {
$options = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => true, // return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_USERAGENT => "spider", // who am i
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
);
$ch = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$output = $header["url"];
return $output;
}
I am now trying to get it to work with HTTPS but it does not follow on it stops at the inputted URL.
Is there anything I can do to fix this?
Add the following options:
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_SSL_VERIFYPEER => false,