After investigating the reasons for the high CPU load generated by an application sending many curl requests with 20 parallel PHP-cli scripts, I've found out the main reason is the PROXY option set to the curl handle.
The following code is generating a 0.5-1.5% CPU load at the moment of curl_exec when run without the proxy and 12-25% when using any proxy (I've tried many different ones).
With many threads running many requests at the same time, it soon becomes critical.
I'm using:
Ubuntu 16.04
PHP 7.0.22
curl 7.55.1
Is there a fix or workaround I can apply?
$address = 'xxx';
$port = 'xxx';
$url = 'https://api.ipify.org?format=json';
do {
$ch = curl_init();
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_ENCODING, 'identity');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_PROXY, $address);
curl_setopt($ch, CURLOPT_PROXYPORT, $port);
curl_setopt($ch, CURLOPT_URL, $url);
curl_exec($ch);
curl_close($ch);
sleep(10);
} while (true);
Related
Hello I need some into about this error. I send loop data using curl that data take too much time to send. I set max_execution_time Limit to 900. So why this Request Timeout happen. Is that server problem or php.ini problem.
NOTE: My Proxy is Working and its high speed. I send data in loop. So after 1mint this error show. in title say 500 Internal server error and in body says Request Timeout
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT,'Opera/9.80 (Series 60; Opera Mini/6.5.27309/34.1445; U; en) Presto/2.8.119 Version/11.10');
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
echo $data = curl_exec($ch);
curl_close($ch);
You might change your curl timeout from 10 seconds (like you have it inside your code right now) from
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
to 900 seconds, like this:
curl_setopt($ch, CURLOPT_TIMEOUT, 900);
These numbers are seconds (not 100% sure). Also find according curl documentation here.
CURLOPT_VERBOSE explains, the system takes time for connecting to PROXY before it actually makes a request to a web link.
$ch = curl_init("$url");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_ENCODING, "gzip");
curl_setopt($ch, CURLOPT_PROXY, "$ip");
curl_setopt($ch, CURLOPT_PROXYPORT, "$po");
$response = curl_exec($ch);
$errmsg = curl_error($ch);
$cInfo = curl_getinfo($ch);
curl_close($ch);
I want to execute multiple links without closing curl to reduce latency. Is there any way to keep the PROXY live and do multiple requests to links?
When curl uses a proxy, the connection keeps open until the call of curl_close.
Two important points,
First/initial proxy connection always gonna take time (latency)
When the connection is established, don't close the connection, simply adjust parameters according to the requirements.
For an example:
//$ch = curl_init("$url");
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_ENCODING, "gzip");
curl_setopt($ch, CURLOPT_PROXY, "$ip");
curl_setopt($ch, CURLOPT_PROXYPORT, "$po");
curl_setopt($context, CURLOPT_URL, $url); // Request One
$response = curl_exec($ch);
$errmsg = curl_error($ch);
$cInfo = curl_getinfo($ch);
curl_setopt($context, CURLOPT_URL, $url); // Request Two
$response = curl_exec($ch);
$errmsg = curl_error($ch);
$cInfo = curl_getinfo($ch);
curl_setopt($context, CURLOPT_URL, $url); // Request Three
$response = curl_exec($ch);
$errmsg = curl_error($ch);
$cInfo = curl_getinfo($ch);
curl_close($ch); //Closing after all requests
The first request will take some time in connection establishment to the proxy which depends on multiple factors. After the first request, all requests will be faster and will happen based on actual proxy speed(provided server script >= proxy speed).
I think we can't do the persistent connection. If you know how to do that, please share :-)
I am trying to use curl through a proxy, when I dont use a proxy it works fine but if I use a proxy it returns null, no error messages or anything.
I have already tried other opened questions but their answers are inconclusive.
Here is my code:
<?php
$url = 'http://dynupdate.no-ip.com/ip.php';
$proxy = '122.117.225.161:8080';
//$proxyauth = 'user:pass';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
//curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyauth);
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_HEADER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
var_dump($curl_scraped_page);
?>
The proxy there is a free working proxy (I manually checked), I have also tried it with a private proxy (I used the user:pass authentication method) but still returns null.
Any information will be helpful.
Thanks!
Modern browsers support HTTPS proxies that can be connected to via a PAC file (see https://www.igvita.com/2012/06/25/spdy-and-secure-proxy-support-in-google-chrome/ if you're not familiar).
I'm trying to replicate the same thing and connect to such a proxy via php CURL, but I'm simply getting a blank response, no headers or content.
My code is as follows:
$url = "http://checkip.dyndns.com";
$proxy = "proxy.domain.com:443";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL , 1);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'test:test');
curl_setopt($ch, CURLOPT_PROXYTYPE, "CURLPROXY_HTTP");
$response = curl_exec($ch);
curl_close($ch);
echo $response;
Any ideas?
If anyone is interested, the solution to this is to use spdycat instead of curl: https://github.com/tatsuhiro-t/spdylay
There's no support for connecting to a proxy with HTTPS with curl yet. There is a work-in-progress branch in git though: https://github.com/bagder/curl/tree/HTTPS-proxy
We will appreciate help to get that in shape to merge.
I'm working on a PHP script right now which sends requests to our school's servers to get real-time information about class sizes for different courses. The script runs perfectly fine when I don't use a proxy, returning a string full of course numbers and available seats. However, I want to make this a service for the students, and I'm afraid that if I make too many requests my ip will get blocked. So I'm attempting to do this through a proxy, with no success. As soon as I add the CURLOPT_HTTPPROXYTUNNEL and CURLOPT_PROXY fields to my requests nothing gets returned. I'm not even sure how to troubleshoot it at this point since I'm not getting an error message of any kind. Does anyone know what's going on or how to at least troubleshoot it?
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$proxy = explode(':', $proxy);
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'tempcookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'tempcookie.txt');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_REFERER, $ref);
$exec = curl_exec($ch);
echo curl_error($ch);
print_r(curl_getinfo($ch));
echo $exec;
Proxy used for tests: 75.147.173.215:8080
I use the following code if I have to use a proxy with curl:
$proxy = "127.0.0.1:8080"; // or something like that
if($proxy !== null){
// no need to specify PROXYPORT again
curl_setopt($ch, CURLOPT_PROXY, $proxy);
// to make the request go through as though proxy didn't exist
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
}
You can set CURLOPT_STDERR and CURLOPT_VERBOSE curl options to save your errors in a file. Also, you may use curl_error() function. BTW, by default, curl should show all errors in STDERR.
Besides, for general check, you can simply specify selected proxy in you browser configuration properties, try to open specific service in browser and see, whether correct response is returned.
UPDATE:
CURLOPT_HTTPPROXYTUNNEL is used to make curl call CONNECT HTTP method when requesting proxy server (see here for details). I tested code without this option - it worked successfully.
Code I used:
$proxy = "75.147.173.215:8080";
$proxy = explode(':', $proxy);
$url = "http://google.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_HEADER, 1);
$exec = curl_exec($ch);
echo curl_error($ch);
print_r(curl_getinfo($ch));
echo $exec;
Here is a well tested function which i used for my projects with detailed self explanatory comments
There are many times when the ports other than 80 are blocked by server firewall so the code appears to be working fine on localhost but not on the server , try using port 80 proxies
function get_page($url){
global $proxy;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
//curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_HEADER, 0); // return headers 0 no 1 yes
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // return page 1:yes
curl_setopt($ch, CURLOPT_TIMEOUT, 200); // http request timeout 20 seconds
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // Follow redirects, need this if the url changes
curl_setopt($ch, CURLOPT_MAXREDIRS, 2); //if http server gives redirection responce
curl_setopt($ch, CURLOPT_USERAGENT,
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt"); // cookies storage / here the changes have been made
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // false for https
curl_setopt($ch, CURLOPT_ENCODING, "gzip"); // the page encoding
$data = curl_exec($ch); // execute the http request
curl_close($ch); // close the connection
return $data;
}