Does anybody know why could cURL under php5 be so damn slow to fail even at 45s timeout, downloading a few kb file on a speedO'light server?
The code is here as requested (although I upped the timeouts even more for the script not to fail during execution and changed useragent to Mozilla/4.0 from initial Chrome):
$ch = curl_init('http://www.somesite.com/' . $key);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.somesite.com/somereferer/');
// curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.39 Safari/530.5');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0');
curl_setopt($ch, CURLOPT_TIMEOUT, 120);
curl_setopt($ch, CURLOPT_DNS_CACHE_TIMEOUT, 600);
hmm, could be a few things, maybe some verbose output will have an error of some kind
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_VERBOSE, true); // some output will go to stderr / error_log
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
$response = curl_exec($ch);
$errStr = curl_error($ch);
$errNum = curl_errno($ch);
$head = curl_getinfo($ch, CURLINFO_HEADER_OUT);
$ci = curl_getinfo($ch);
print_r(array($head, $errStr, $errNum, $ci));
Sometimes the user agent will change how a site responds, may need to do something like:
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.001 (windows; U; NT4.0; en-US; rv:1.0) Gecko/25250101');
When I set a CONNECTtimeout, I get faster response.
Including this option:
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT,1)
Related
I'm trying to perform a web request through a SOCKS5 proxy with authentication, but it's not working and I can't work out why. Here's my code:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5_HOSTNAME);
curl_setopt($ch, CURLOPT_PROXY, 'host:port');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'user:pass');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
return curl_exec($ch);
When I use curl_getinfo($ch, CURLINFO_HTTP_CODE), it says the response code is 0. Am I missing something obvious here? The proxy works fine with other programs I use, so I know it isn't an issue with the proxy.
I am getting curl http code as 403:
The user myuser was denied access to perform the operation on the object defect due to the following reasons: The permission setting for operation Create doesn't allow user to perform the required operation on the secured object.
Same code was working fine with 'HTTP', now it is 'HTTPS', so I updated code like added curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); and curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); these lines and it is giving 403 as HTTP CODE. Can someone help me where I am wrong.
$ch = curl_init(MY_BASE_URL);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13');
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_COOKIEFILE, MY_COOKIES);
curl_setopt($ch, CURLOPT_COOKIEJAR, MY_COOKIES);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: application/json; charset=utf-8","Accept:application/json, text/javascript, */*; q=0.01"));
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($myfields));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response=curl_exec($ch);
$code=curl_getinfo($ch, CURLINFO_HTTP_CODE);
if($code!=201)
{throw new Exception("Error creating defect: ".$response);}
else{echo 'Success';}
try this user agent(if not works then try with random user agents)
$useragent= "curl/7.39.0";
curl_setopt($ch,CURLOPT_USERAGENT, $useragent);
I'm trying to replicate the functionality of a PHP 'header' redirect using cURL so that it works within a cronjob triggered script.
The following code runs within a while loop (which I know works) but it returns a 500 internal server error when I try to run it, as though the request is timing out on the server (it should not take any more than 30 seconds).
Is there something wrong with the way I'm doing this:
<?php
$url = 'http://exampleurl/?ld_LeadProductType=PROF&ld_contactFirstName='.rawurlencode($quotes['name']).'&ld_businessName='.rawurlencode($quotes['company_name']).'&QuotePrice=%20/%20Quote%20Price:'.number_format($quotes['quote'],2).'.';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT ,0);
curl_setopt($ch, CURLOPT_TIMEOUT, 128);
$html = curl_exec($ch);
$redirectURL = curl_getinfo($ch,CURLINFO_EFFECTIVE_URL );
curl_close($ch);
?>
I am trying to download a pdf using cURL and am getting stuck on a "Server Error in '/' Application" page. My code:
$url = "https://some.domain.com/Reports/Report?ReportID=123456"
$ch = curl_init($url);
$header = array ('Host: some.domain.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSLVERSION, 4);
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, implode(':', $arrayCiphers));
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_COOKIE, "ASP.NET_SessionId=XXXXXXXX; __RequestVerificationToken_XXXXXXX=lots-of-alpha-numeric-characters");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_REFERER, "https://some.domain.com");
$output = curl_exec($ch);
echo $output;
curl_close($ch);
Is there something else I can try or some more debugging I can do?
[edit] Apparently it's caused by one of my parameters. There are several parameters in the URL. &Flag=True seems to be causing the error. If I change it to &Flag=False I get a blank page.
This error was being caused because my curl request header did not include cookies that were required to download the file. Adding those cookies in the header fixed it.
I've been playing with this curl facebook login script for a while just trying to get to grips with some of the features in curl, but it seems that I can not get the cookies to register:
php script
function facebookLogin(){
$login_email = 'email';
$login_pass = 'pass';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.facebook.com/login.php');
curl_setopt($ch, CURLOPT_POSTFIELDS,'email='.urlencode($login_email).'&pass='.urlencode($login_pass).'&login=Login');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3");
curl_setopt($ch, CURLOPT_REFERER, "http://www.facebook.com");
$page = curl_exec($ch);
echo $page;
}
I have a text file called cookies.txt which is in the same directory as the script, but after running this script nothing is written into the file and therefore no cookies are created, this is a big issue when trying to explore other web pages on the same website as you have to keep logging in.
Where am I going wrong?
Ok it turns out it is registered even if the cookies.txt file is empty but you need to make sure you call this file when you try to explore other parts of the site e.g.
function facebookGoToMessages(){
facebookLogin();
$ch = curl_init ("http://www.facebook.com/messages");
curl_setopt ($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3");
curl_setopt($ch, CURLOPT_REFERER, "http://www.facebook.com");
$page = curl_exec ($ch);
echo $page;
}