How to resend curl request if the output is empty or something like (Database maintenance, please check back in 10 minutes) Want to refresh request if it is similar to any above.
Im using basic curl method
$this->ch = curl_init();
curl_setopt($this->ch, CURLOPT_HEADER, true);
curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, true);
#curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($this->ch, CURLOPT_MAXREDIRS, 10);
curl_setopt($this->ch, CURLOPT_USERAGENT, 'Opera/9.23 (Windows NT 5.1; U; en)');
// site is returning a gzipped body, force uncompressed
curl_setopt($this->ch, CURLOPT_ENCODING, 'identity');
If this is very simple question, Im really sorry Because im new.
Thanks in advance
If you want refresh it immediately in same script execution you can do something like that:
$response = curl_exec($this->ch);
if (empty($response)) {
sleep(10);
$response = curl_exec($this->ch);
}
Or combination of that in some loop if you want check like 3 times or more.
Related
The following curl-call succeeds every time, if and only if $data is printed after the curl-call. curl_getinfo() returning
[content_type] => text/html; charset=UTF-8
If $data is not printed, the curl-call sometimes return the same result as above and sometimes returns $data being "Loading...", Which means that page has not finished loading yet. And curl_getinfo() returning
[content_type] => text/html
Furthermore, when using print_r($data), I can see the print_r(curl_getinfo($ch)); on my website being updated several times while performing the curl-call. What... The.... F?
(the set_opt-list has grown larger as I'm trying to find a solution LOL)
Ooh.. yeah, even if I print $data after it's been returned to function caller and caught in another variable.. curl succeeds every time.
Is this normal behaviour? I don't want to print_r($data)!
Is it possible that the url I'm retrieving contains javascript which gets run when I "print" it on my website? Why does it work occasionally without the print_r($data)? Ref: is-there-a-way-to-let-curl-wait-until-the-pages-dynamic-updates-are-done
edit: Until further notice, I've put the curl-call in a while-loop, checking if downloaded size is above a certain threshold. I've set the while loop to 10 iterations, and so far it is enough, i.e. it will manage to download the content of interest. Time consumed is barely noticed.
function curl_get_contents($url) {
global $dbg;
$ch = curl_init();
$timeout = 30;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_NOSIGNAL, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
//curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17');
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch,CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
$data = curl_exec($ch);
if ($dbg) {
print_r(curl_getinfo($ch)); // This one gets refreshed if print_r($data) used below
if(curl_errno($ch)){
echo 'Curl error: ' . curl_error($ch);
} else {
echo "ALL GOOD <br>";
}
}
curl_close($ch);
//echo $data; // If I do this...
//print_r($data); // ... or this. curl is success 100%.
return $data;
}
I'm trying to access remote files on a website that requires me to log in to it. The current way I'm doing this is with a wget ran from shell_exec. This 'works' but is definitly not the way I want to do it in. I would prefer to do it using cURL, but for some reason this will not work for me.
What happens instead of getting the page I requested is I'm redirected to the login page, which does not happen with the wget method, which should be making the exact same request...
What am I doing wrong?
First of all this is the current (not nice but) working method of accessing the page:
shell_exec("wget --post-data='serviceLoginUser=something&serviceLoginPass=something&qq=login' -p https://internal.website.com/somepage.php?get=request -O return.json");
This is my download function which does not work but I want to replace the old way with: (updated after suggestion from Abkarino)
public function download($url, $postData = NULL){
//Set cURL options
curl_setopt($this->ch, CURLOPT_URL, $url);
curl_setopt($this->ch, CURLOPT_HEADER, 0);
//To act like a normal browser (needed for intranet)
curl_setopt($this->ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17');
curl_setopt($this->ch, CURLOPT_AUTOREFERER, true);
curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($this->ch, CURLOPT_VERBOSE, 1);
//TODO: remove
curl_setopt($this->ch, CURLOPT_SSL_VERIFYPEER, false);
//If POST data was supplied:
if ($postData != NULL){
curl_setopt($this->ch,CURLOPT_POSTFIELDS, $postData);
}
//execute post
return curl_exec($this->ch);
}
Where the following code will form the request: (currenly this code is only to test if the cURL actually returns the correct data)
include 'Downloader.php';
$dl = new Downloader;
$postData = array(
'serviceLoginUser' => 'something',
'serviceLoginPass' => 'something',
'qq' => 'login'
);
echo $dl->download("https://internal.website.com/somepage.php?get=request", $postData);
I want to check the links in my database, if the status of the link (through possible redirections) is still valid (e.g. status 200). The below script is what I currently use. The limitation is that over +/- 400 links, the server gives me a 500 - internal error. Unfortunality, I cannot review the servers logs on what the reason is, my assumption is that it's a time out issue.
How can I make this script scalable, so that it will allow me to run more then the currently +/- 400 links?
function urlValidator($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 30);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
$data = curl_exec($ch);
$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
if ($httpcode != '200') {
echo $url;
echo " - ". $httpcode;
}
}
// creation of $url_array
//
foreach($url_array as $url){
if(!is_null($url)) {
urlValidator($url);
}
}
I did try to add flush() and/or ob_flush() to the code, but it didn't help either (or implemented wrongly).
Any suggestions are more then welcome.
The default execution time of a PHP script is 30 seconds. After that it will time out.
You can either increase this time to something like this:
ini_set('max_execution_time', 600); //10 minutes
But, to make it really scalable, I would store the current "link-check" status in a database, so that you can continue where you left off and have multiple instances call your script.
I'm using cURL in PHP to run a script that can take over an hour to run. For debugging purposes I'd like to be able to view the progress of the request by looking at the screen (this is not a public site) and see what's going on. I've tried a few things but am not having any luck. I don't need a lot of info, just something like 'now loading id 123'
I've tried ob_flush, but apparently that is no longer supported as per: http://php.net/ob_flush
I've also tried using CURLOPT_PROGRESSFUNCTION but there is not a lot of documentation on it and I couldn't get it to work. My code is very simple:
$sql = "select item_number from products order by id desc";
$result_sql = $db->query($sql);
while($row = $result_sql->fetch_assoc())
{
//I'd like it to display this as it loads
print '<br>Getting data for item_number: '.$row['item_number']
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL,"http://targetsite.com//".$row['item_number']);
curl_setopt($curl,CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_SSLVERSION, 3);
//curl_setopt($curl, CURLOPT_HEADER, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($curl, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 ( .NET CLR 3.5.30729)");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_VERBOSE, 1);
}
Any advice? I'm really not fussy, just the easiest method that will quickly show me that something is happening.
Assuming you're using PHP >=5.3 (previous versions do not support CURLOPT_PROGRESSFUNCTION) you use it like so:
function callback($download_size, $downloaded, $upload_size, $uploaded)
{
// do your progress stuff here
}
$ch = curl_init('http://www.example.com');
// This is required to curl give us some progress
// if this is not set to false the progress function never
// gets called
curl_setopt($ch, CURLOPT_NOPROGRESS, false);
// Set up the callback
curl_setopt($ch, CURLOPT_PROGRESSFUNCTION, 'callback');
// Big buffer less progress info/callbacks
// Small buffer more progress info/callbacks
curl_setopt($ch, CURLOPT_BUFFERSIZE, 128);
$data = curl_exec($ch);
Source: http://pastebin.com/bwb5VKpe
Trying to send data to a server which accepts data in the following format
VERIFY_DATA=MER_ID=xxx|MER_TRNX_ID=xxx| MER_TRNX_AMT=xxx
Will the following lines do?
$datatopost="VERIFY_DATA=MER_ID=xxx|MER_TRNX_ID=xxx| MER_TRNX_AMT=xxx";
curl_setopt ($ch, CURLOPT_POSTFIELDS,$datatopost);<br />
Any help will be appreciated,I am new with curl.
you can use this article to see how to do it properly.
I personally used this code to do it on a project of mine
$data="from=$from&to=$to&body=".urlencode($body)."&url=$url";
//$urlx contains the url where you want to post. $data contains the data you are posting
//$resp contains the response.
$process = curl_init($urlx);
curl_setopt($process, CURLOPT_HEADER, 0);
curl_setopt($process, CURLOPT_POSTFIELDS, $data);
curl_setopt($process, CURLOPT_POST, 1);
curl_setopt($process, CURLOPT_RETURNTRANSFER,1);
curl_setopt($process,CURLOPT_CONNECTTIMEOUT,1);
$resp = curl_exec($process);
curl_close($process);
here is a working example in php. This asks for and returns an FX quote. My data request is in the URL yours is in the post-fields though so you need to adjust. It looks as though you have spaces in the data you are passing "x| ME" i suspect it will not like that.
$ch = curl_init(); // initialise CURL
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $curl_opt_string ); // this contains the URL and the request
curl_setopt($ch, CURLOPT_HEADER, false); // no header
curl_setopt($ch, CURLOPT_INTERFACE, "93.129.141.79"); // where to send the data back to / outgoing network
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"); // tell it what the agent is
curl_setopt($ch, CURLOPT_POST, 1); // want the data back as a post
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // return as a string rather than to the screen
$output = curl_exec($ch); // varu=iable to return it to
curl_close($ch); // close cURL resource, and free up system resources
$subject = $output; // get the data
I constructed my output sting as follows
$curl_opt_string = "http://msxml.rexefore.com/index.php?username=MiJoee4r65&password=L8e44Y&instrument=245.20." . $lhsrhs . "LITE&fields=D4,D6";