I am trying to test the below function but every time I try to use any sort of proxy IP (I have tried about 15 now) - I generally get the same error:
Received HTTP code 0 from proxy after CONNECT
Here is the function, anything wrong with it? It could just be the proxies I am using but I have tried several times now.
function getPage($proxy, $url, $referer, $agent, $header, $timeout) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, $header);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
$result['EXE'] = curl_exec($ch);
$result['INF'] = curl_getinfo($ch);
$result['ERR'] = curl_error($ch);
curl_close($ch);
return $result;
}
Also in general, anyway I can improve it?
I appreciate all help.
Update
As I submitted this, I tried another proxy and it worked!
The other question still stands, how can I improve the above. It takes about 3-4 seconds to execute, anything I can do, or is this too minimal?
I know you sort of answered your first problem but code 0 is not a valid http status code. They should all begin with either 1 (informational), 2 (success), 3 (redirection), 4 (client error), or 5 (server error). I would be really interseted if anyone knows why you might get this code. Searching the libcurl site didn't bring anything up.
(More detailed information is here if you are interested:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.htmlt)
For the second question I think you would need to find where the longest operation was.The microtime() function might be useful to you here. The documentation for microtime() has some example scripts to help you use the timer.
I suspect though that most of the 3-4 seconds could be waiting to get the response via the proxy at curl_exe($ch).
Related
i have an array with approximately 45 k usernames in in i want to hit a url using curl that would give me a response pertaining to those usernames.The issue is i want to achieve it in less time.
$username=['123','456','789'....] //upto 45k entries
for($i=0;$i<sizeof($username);$i++)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://abc.com.pk/hxc/get_user_details.php?uname='.$username[$i]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_USERAGENT, $ua);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 20);
curl_setopt($ch, CURLOPT_HTTPGET, true);
$result = curl_exec($ch);
curl_close($ch);
}
The above code depicts what i am doing right now but as usernames are in large numbers it takes alot of time to return all the responses.Is there any way i can achieve it in less time.
Have a look at https://github.com/php-curl-class/php-curl-class , it speeds up our curl requests a lot.
It has multi-curl support enabled and it's very easy to use.
As for your question on the time, you can set the time out using
Curl::setTimeout($seconds)
Or in the case of MultiCurl
MultiCurl::setTimeout($seconds)
You can extend the timeout as much time as needed.
You can use curl-multi-init and curl-multi-exec so that your requests are processed asynchronously.
I'm currently making a PHP site that gets his data from an API. At first cURL seemed to do that perfectly, but if the API returns an empty response (and we make the request again, because we can't give a correct response without it) it seems to spawn a child process. This didn't use too much CPU when developing, but in production it can get as high as 150% CPU load.
Code used to get data from the API:
while (empty($output )) {
$ch = curl_init();
set_curl($ch);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 8);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 4);
curl_setopt($ch, CURLOPT_URL, "http://domain.com");
$output = curl_exec($ch);
}
Is there any way to fix this?
I have a strange problem which may or may not be tied to Plesk. This PHP script intends to fetch a page on the same server when executed on the same domain e.g. http://quotationsbook.com/sometestpage.php
<?php
error_reporting('E_ALL');
ini_set('display_errors', 1);
function curlFileGetContents($urlreq) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 50);
curl_setopt($ch, CURLOPT_URL, $urlreq);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$request_result = curl_exec($ch);
if (curl_errno($ch))
$response = 0;
else
$response = $request_result;
curl_close($ch);
return $response;
}
$url = 'http://quotationsbook.com';
$data = curlFileGetContents($url);
echo '<pre>';
print_r($data);
echo '</pre>';
?>
However, it does not fetch the page requested, it always fetches a Plesk error page.
In terms of the PHP var allow_url_fopen it is set to On.
The problem only occurs for the same domain where the code is hosted, not for other domains. i.e. it only occurs on http://quotationsbook.com where what I'm trying to fetch is under http://quotationsbook.com/*, it does not occur when I try to fetch say, http://google.com
The answer to your question is actually in your last paragraph.
The problem only occurs for the same domain where the code is hosted,
not for other domains. i.e. it only occurs on
http://quotationsbook.com where what I'm trying to fetch is under
http://quotationsbook.com/*, it does not occur when I try to fetch
say, http://google.com
That is your clue.
I must pose a question and that is why use CURL if you are on the same server? What are you trying to accomplish? There are probably better solutions out there than using CURL.
Assuming you need to use CURL, it's likely a firewall or other security issue. Make sure the port being accessed is open and not blocked.
See PHP Curl does not work on localhost?
The API documentation clearly states that the wp.newPost returns the new post id, but roughly 40% of the time we're getting a response of false, but with and HTTP Response Code of 100. The post is published to Wordpress successfully either way.
$ch = curl_init();
curl_setopt($ch, CURLOPT_POSTFIELDS, $request);
curl_setopt($ch, CURLOPT_URL, $this->getXMLRPCURL());
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 1);
// Encode htaccess credentials if they're set
$htaccess_credentials = $this->getHtaccessCredentials();
if ($htaccess_credentials) {
curl_setopt($ch, CURLOPT_USERPWD, $htaccess_credentials->toString());
}
$results = curl_exec($ch); // <<-- RETURNS FALSE HERE.
$http_status = curl_getinfo($ch, CURLINFO_HTTP_CODE); // Returns 100
curl_close($ch);
return $results;
I'm happy to post a sample $request string, but that part seems to be working fine. I was just wondering if anyone has run into this before? It seems to have been run into here, but with little resolution.
Sheesh. Timeout. Duh. Someone had set the CURL timeout to 1. Nonetheless, we're still not receiving full responses sometimes even without a timeout set - likely the default socket timeout is kicking in. Nonetheless, we have a (somewhat weighty) workaround.
Have a nice day.
I know how to set the timeout in cURL but I want to alert the user that the request timed out.
I have created an ajax script that allows the user to request data from various insurance sites and aggregate into a list. If any of the insurance sites fail to respond within a certain time I want to alert the user that the current quote from that company is not available at the moment.
Does cURL return anything to signal a timeout?
curl_errno() returns 28 if the operation timed out. See http://curl.haxx.se/libcurl/c/libcurl-errors.html for other error codes.
Or another solution that can cover even more cases (server timed out, server errored out with a blank page) is to check if your get_url function result is different that "" or FALSE.
Example of get_url function :
function get_url($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$tmp = curl_exec($ch);
curl_close($ch);
return $tmp;
}