I am using curl and setting all the parameters correctly (as far as I know) but CURLOPT_TIMEOUT is being ignored and allowing for an infinite loop. Here is the configuration for my Curl Request:
$user_agent = 'Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';
$options = array(
CURLOPT_CUSTOMREQUEST => "GET", //set request type post or get
CURLOPT_POST => false, //set to GET
CURLOPT_USERAGENT => $user_agent, //set user agent
CURLOPT_COOKIEFILE => dirname(__FILE__)."/cookie.txt", //set cookie file
CURLOPT_COOKIEJAR => dirname(__FILE__)."/cookie.txt", //set cookie jar
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_SSL_VERIFYPEER, FALSE, //ignore ssl
CURLOPT_PROXY =>$proxy['ip'],
CURLOPT_PROXYPORT =>$proxy['port'],
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_AUTOREFERER => true, // set referrer on redirect
CURLOPT_CONNECTTIMEOUT => 20, // timeout on connect
CURLOPT_TIMEOUT => 10, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
);
$ch = curl_init($url);
curl_setopt_array($ch, $options);
$content = curl_exec($ch);
$err = curl_errno($ch);
$errmsg = curl_error($ch);
$header = curl_getinfo($ch);
curl_close($ch);
I am not the best at debugging so I'm not sure what the problem could be. Please help me.
The curl_setopt($connection, CURLOPT_TIMEOUT, $seconds) should be called just right before curl_exec() function.
It wasn't working for me when I called it sooner
If your are using CloudFlare note that :
Enterprise customers can increase the 524 timeout up to 6000 seconds
using the proxy_read_timeout API endpoint. If you regularly run HTTP
requests that take over 100 seconds to complete (for example large
data exports), move those processes behind a subdomain not proxied
(grey clouded) in the Cloudflare DNS app.
Related
I've got a problem with my PHP script using CURL. I want to download file from another server (Probably with hotlink protection). The URL is like this: example.com/script.extension?id=12345&type=xxx&another=qwerty
When i hit that link from my website, It sends 'Referer:' in headers, so the download is not starting. (without the header it works)
I have to modify the file's name, so i've used CURL to download that to my server. When I'm doing that with that script, the file is created, but it is empty:
function my_function($the_url) {
$fp = fopen ('tmp_file.extension', 'w+');
$user_agent='Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';
$options = array(
CURLOPT_VERBOSE => 1,
CURLOPT_URL => $the_url,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_POST => false,
CURLOPT_USERAGENT => $user_agent,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => "",
CURLOPT_AUTOREFERER => false,
CURLOPT_REFERER => "http://example.com/",
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
);
$ch = curl_init();
curl_setopt_array($ch, $options);
$my_exec = curl_exec($ch);
fwrite($fp, $my_exec);
curl_close($ch);
fclose($fp);
}
But it works well with file which is hosted on my site (eg. mysite.com/file.zip)
Also, when I run command through the shell:
$ curl -L -o file.tmp --referer http://example.com/ full url here
It downloaded the file properly.
What's the reason I cannot do it with my PHP function?
This is the php code that I am using.
I tested this code on localhost (xampp) and everything ok. but when I upload this code to my host (1and1 hosting) This code will not work. Please help me to find out why?
Thank you very much!
$url = "https://www.packagetrackr.com/track/1ZX799390355046642";
$page = get_web_page($url);
echo $page;
function get_web_page($url)
{
$options = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => true, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_USERAGENT => "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5", // who am i
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
CURLOPT_SSL_VERIFYPEER => false, // Disabled SSL Cert checks
CURLOPT_REFERER => $url // referent
);
$ch = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
curl_close( $ch );
return $content;
}
you can try turn on allow_url_fopen in your php.ini file
or
php_value allow_url_fopen On in .htaccess
I am using curl to hit a url and get contents from it and below is the code
$url = 'http://uber.com';
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_URL => $url,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.16 (KHTML, like Gecko) \ Chrome/24.0.1304.0 Safari/537.16',
CURLOPT_CONNECTTIMEOUT => 180,
CURLOPT_TIMEOUT => 180 + 60,
CURLOPT_COOKIE => 'lang=en;'
));
$response = curl_exec($ch);
var_dump($response);
It gives me the content when I execute the script on local server but while executing this on staging and production server, situated in US, it give 403-forbidden error. I also tested the same from another server in US and it works fine.
Any help is appreciated.
I am trying to scrape a website using PHP, CURL and POST method in order to submit a form before web scraping the page. The problem I am experiencing is that there is connected with POST method: no data is submitted to the server, so the scraped webpage doesn't contain what I am looking for.
I quit sure the problem is connected with the form type: enctype="multipart/form-data".
How can I manage this POST request, considering that the form is multipart/form-data?
Do I have to encode the post_string in a special way?
Here's the code I'm using:
function curl($url) {
//POST string
$post_string="XXXX";
$options = Array(
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_AUTOREFERER => TRUE,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8",
CURLOPT_URL => $url,
CURLOPT_CAINFO => dirname(__FILE__)."/cacert.pem",
CURLOPT_POSTFIELDS => $post_string,
);
$ch = curl_init();
curl_setopt_array($ch, $options);
$data = curl_exec($ch);
curl_error($ch);
curl_close($ch);
return $data;
}
$scraped_page = curl("XXXURLXXX");
echo $scraped_page;
Thank you!
Set the CURLOPT_POST to true:
CURLOPT_POST = true
Then fill your post fields like this 'setup':
$postfields = array();
$postfields['field1'] = 'value1';
$postfields['field2'] = 'value2';
CURLOPT_POSTFIELDS => $postfields
If value is an array, the Content-Type header will be set to multipart/form-data.
The PHP manual
Yes, $post_string needs to be an array.
Also set CURLOPT_POST to true.
I am developing one SMS application. I have to call one link
for eg http://sendsms.com/send.php?mobile=45455&msg=hello for sending SMS
so I used CURL concept to send sms.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
$options = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 15, // timeout on connect
CURLOPT_TIMEOUT => 15, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
);
curl_setopt_array($ch, $options);
$content = curl_exec($ch);
$err = curl_errno($ch);
$errmsg = curl_error($ch);
$header = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
$output ="Error:".$err." ErrMsg:".$errmsg." Header:". json_encode($header);
The Problem is message is not sending. I checked the SMS API server. And There also no request is received. So the only problem is CURL may not calling the server. I tried to print the Error msg and I got curl_errno($ch) is 0
please provide me the best way to do this
Try to send request by POST:
curl_setopt($ch, CURLOPT_POST ,1);