CURLOPT_TIMEOUT , is there "else" function? - php

<?php
function get_random_proxy()
{
srand ((double)microtime()*1000000);
$f_contents = file ("proxy.txt");
$line = $f_contents[array_rand ($f_contents)];
return $line;
}
$proxy = get_random_proxy();
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "example.com");
curl_setopt($ch, CURLOPT_PROXY,$proxy);
curl_setopt($ch, CURLOPT_TIMEOUT ,30);
curl_exec($ch);
curl_close($ch);
?>
curl will close connection if can not connect within the 30 seconds.
as you can see, i'm using proxy list. however, some proxy ips sometimes have problems to connect within the 30 seconds, and curl is closing connection when can not load in 30 seconds.
i wanna try another ip for curl connect if curl timeout reached. right now, curl is closing everything if ip isn't working. i wanna try another ip. well, could you please suggest me a function?
edited for #rubayeet. added new proxy function

You just have to use curl_errno to test if CURLE_OPERATION_TIMEDOUT occured

function get($url, $proxy){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY,$proxy);
curl_setopt($ch, CURLOPT_TIMEOUT ,30);
$response = curl_exec($ch);
curl_close($ch);
return $response
}
$url = 'example.com';
while(true) {
$proxy = get_random_proxy();
$response = get($url, $proxy);
if ($response === False) continue;
else break;
}
//do something with $response

You have to create a new curl session to connect to another proxy. So put a foreach loop around your code and loop through your proxy array.
Also you can use curl_errno() and curl_error() to check for an error (like your timeout).
Maybe it would be useful to set CURLOPT_RETURNTRANSFER and load it into a var to modify or work on it.

Related

PHP cURL Received HTTP code 407 from proxy after CONNECT

I have a php curl with proxy problem.
Below is my code:
<?php
$proxylist = file('proxy.txt');
$random_proxy = $proxylist[mt_rand(0,count($proxylist)-1)];
$pinfos = explode(':', $random_proxy);
$proxyipport = $pinfos[0].':'.$pinfos[1];
$proxyuserpwd = $pinfos[2].':'.$pinfos[3];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,'https://google.com');
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);
curl_setopt($ch, CURLOPT_PROXY, $proxyipport);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyuserpwd);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($ch);
if (curl_errno($ch)) {
echo 'Error: ' . curl_error($ch).'<br/>';
}
curl_close($ch);
echo $result;
?>
The format of proxies in proxy.txt is ip:port:user:pass and all proxies are working.
The problem is when I used $proxyipport and $proxyuserpwd in CURLOPT_PROXY and CURLOPT_USERPWD, the curl result threw the error Received HTTP code 407 from proxy after CONNECT. However, when I replaced those variables with actual ip:port, user:pass, it worked as normal. I also did an echo of $proxyipport and $proxyuserpwd and it showed me the exact ip:port and user:pass as expected.
Can someone please tell what I did wrong and how to fix that?
Thanks in advance!
Most likely it is the newline \n, so try:
$proxylist = file('proxy.txt', FILE_IGNORE_NEW_LINES);
If it is another hidden character(s) or a Windows format file then with \r you can try:
$pinfos = explode(':', $random_proxy);
$pinfos = array_map('trim', $pinfos);

Icecast metadata update by php

I need to get Icecast metadata auto update by PHP let say every 15 min which will be done by cPanel cronjob.
i had the below code but it does't work(it works if I use header location to redirect however cronjob won't be able to do that)
<?PHP
$url="http://tgftp.nws.noaa.gov/data/observations/metar/stations/KJFK.TXT";
$info=file_get_contents($url);
$url_info = "http://username:password#icecast:8000/admin/metadata?mount=/mymount&mode=updinfo&song=" . urlencode($info);
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url_info);
curl_setopt($ch, CURLOPT_HEADER, 0);
// grab URL and pass it to the browser
curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
?>
Try checking for errors after you execute the call using curl_error:
<?php
$url="http://tgftp.nws.noaa.gov/data/observations/metar/stations/KJFK.TXT";
$info=file_get_contents($url);
$url_info = "http://username:password#icecast:8000/admin/metadata?mount=/mymount&mode=updinfo&song=" . urlencode($info);
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url_info);
curl_setopt($ch, CURLOPT_HEADER, 0);
// grab URL and pass it to the browser, check for errors
if (curl_exec($ch) === FALSE)
{
print 'Curl-Error occurred: ' . curl_error($ch).', error code: '.curl_errno($ch);
}
// close cURL resource, and free up system resources
curl_close($ch);

curl http code with redirections

I'm new with curl and I can't find my answer.
I want to get the http status of a page, so I'm using
$httpCode = curl_getinfo($curl, CURLINFO_HTTP_CODE);
But this does not work if there is a redirection (for example a 301 that redirect to another page, it will give me a 200 answer) because curlinfo_http_code gives the Last received HTTP code
Any idea how I can get the first received http code ?
thanks
You need to configure curl with CURLOPT_FOLLOWLOCATION = 0
The way I am accomplishing this is by having PHP consult an INI file with the different return codes and replacing them with something human readable
<?php
$ch = curl_init(); // create cURL handle (ch)
if (!$ch) {
die("Couldn't initialize a cURL handle");
}
// set some cURL options
$ret = curl_setopt($ch, CURLOPT_URL, "http://mail.yahoo.com");
$ret = curl_setopt($ch, CURLOPT_HEADER, 1);
$ret = curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$ret = curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
$ret = curl_setopt($ch, CURLOPT_TIMEOUT, 30);
// execute
$ret = curl_exec($ch);
if (empty($ret)) {
// some kind of an error happened
die(curl_error($ch));
curl_close($ch); // close cURL handler
} else {
$info = curl_getinfo($ch);
curl_close($ch); // close cURL handler
if (empty($info['http_code'])) {
die("No HTTP code was returned");
} else {
// load the HTTP codes
$http_codes = parse_ini_file("HTMLCodes.ini");
// echo results
echo "The server responded: <br />";
echo $http_codes[$info['http_code']];
}
}
?>
example of the ini file with return codes
[Informational 1xx]
100="Continue"
101="Switching Protocols"
[Successful 2xx]
200="OK"
201="Created"
202="Accepted"
203="Non-Authoritative Information"
204="No Content"
205="Reset Content"
206="Partial Content"
[Redirection 3xx]
300="Multiple Choices"
301="Moved Permanently"
302="Found"
303="See Other"
304="Not Modified"
305="Use Proxy"
306="(Unused)"
307="Temporary Redirect"

Asynchronous cURL using POST

I am making a command line application. I need to send out multiple POST requests via cURL simultaneously after I have performed log in procedures - meaning outgoing requests must send session id etc.
The chain of events is as follows:
I open cURL connection with curl_init
I log in to remote site sending POST request with curl_exec and get returned HTML code as response
I send multiple POST requests to same site simultaneously.
I was thinking of using something like that:
// Init connection
$ch = curl_init();
// Set curl options
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, 1);
// Perform login
curl_setopt($ch, CURLOPT_URL, "http://www.mysite/login.php");
$post = array('username' => 'username' , 'password' => 'password');
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post));
$result = curl_exec($ch);
// Send multiple requests after being logged on
curl_setopt($ch, CURLOPT_TIMEOUT_MS, 1);
for($i = 0 ; $i < 10 ; $i++){
$post = array('myvar' => 'changing_value');
curl_setopt($ch, CURLOPT_URL, 'www.myweb.ee/changing_url');
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post));
curl_exec($ch);
}
But this doesn't seem to work as only the first request in loop seems to be sent.
Using curl_multi_init would probably one solution but I don't know if i can pass it the same cURL handle multiple times with changed options for each.
I don't need any response from server for those simultaneous requests but it would be awesome if it also can be done somehow.
It would be perfect if someone could push me in the right direction how to do it.
You'll need to create a new curl handle for every request, and then register it with http://www.php.net/manual/en/function.curl-multi-add-handle.php
here is some code i ripped out and adapted from my code base, have in mind that you should add error checking in there.
function CreateHandle($url , $data) {
$curlHandle = curl_init($url);
$defaultOptions = array (
CURLOPT_COOKIEJAR => 'cookies.txt' ,
CURLOPT_COOKIEFILE => 'cookies.txt' ,
CURLOPT_ENCODING => "gzip" ,
CURLOPT_FOLLOWLOCATION => true ,
CURLOPT_RETURNTRANSFER => true ,
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => $data
);
curl_setopt_array($curlHandle , $defaultOptions);
return $curlHandle;
}
function MultiRequests($urls , $data) {
$curlMultiHandle = curl_multi_init();
$curlHandles = array();
$responses = array();
foreach($urls as $id => $url) {
$curlHandles[$id] = CreateHandle($url , $data[$id]);
curl_multi_add_handle($curlMultiHandle, $curlHandles[$id]);
}
$running = null;
do {
curl_multi_exec($curlMultiHandle, $running);
} while($running > 0);
foreach($curlHandles as $id => $handle) {
$responses[$id] = curl_multi_getcontent($handle);
curl_multi_remove_handle($curlMultiHandle, $handle);
}
curl_multi_close($curlMultiHandle);
return $responses;
}
There's a faster, more efficient option ... that doesn't require that you use any curl at all ...
http://uk3.php.net/manual/en/book.pthreads.php
http://pthreads.org
See github for latest source, releases on pecl ....
I will say this, file_get_contents may seem appealing, but PHP was never designed to run threaded in this manner, it's socket layers and the like give no thought to consumption you might find that it's better to fopen and sleep inbetween little reads to conserve CPU usage ... however you do it it will be much better ... and how you do it depends on what kind of resources you want to dedicate the task ...

PHP cURL required only to send and not wait for response

I need a PHP cURL configuration so that my script is able to send requests and ignore the answers sent by the API.
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string);
// curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
//curl_setopt($ch, CURLOPT_TIMEOUT_MS, 100);
$result = curl_exec($ch);
echo $result;
curl_close ($ch);
I tried adding:
// curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
//curl_setopt($ch, CURLOPT_TIMEOUT_MS, 100);
But its not working properly and the API webserver is not receiving the requests.
The reason for this is I am sending large amount of requests to the API therefore my script is very slow because it waits for each and every request.
Any help is appreciated.
Sender file example ./ajax/sender.php
Script sending POST -> it makes full request to host, but it doesn't wait on answer from server : CURLOPT_HEADER(0) we dont needs headers from server) and CURLOPT_RETURNTRANSFER (false) we don't needs data from server.CURLOPT_TIMEOUT - Extra procteted : We waiting after sent only 1ms on respond server, this is extra quaranty to not wait any more ms if server keep us. ### NOTE ### HTTP1.1 has one package max 16kb. HTTP2 has 36kb one pacakge. If POST are more biggest, server will be send with many packages in series = $SIZE%16kb
$url = 'https://127.0.0.1/ajax/received.php';
$curl = curl_init();
$post['test'] = 'examples daata'; // our data todo in received
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt ($curl, CURLOPT_POST, TRUE);
curl_setopt ($curl, CURLOPT_POSTFIELDS, $post);
curl_setopt($curl, CURLOPT_USERAGENT, 'api');
//curl_setopt($curl, CURLOPT_TIMEOUT, 1); //if your connect is longer than 1s it lose data in POST better is finish script in recevie
curl_setopt($curl, CURLOPT_HEADER, 0);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, false);
curl_setopt($curl, CURLOPT_FORBID_REUSE, true);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 1);
curl_setopt($curl, CURLOPT_DNS_CACHE_TIMEOUT, 100);
curl_setopt($curl, CURLOPT_FRESH_CONNECT, true);
curl_exec($curl);
curl_close($curl);
Received file example ./ajax/received.php
ignore_user_abort(true); //if connect is close, we continue php script in background up to script will be end
header("Connection: close\r\n");
header("Content-Encoding: none\r\n");
header("Content-Length: 1");
### we just close connect above if webbrowser, or request waiting on answer ( we know we set CURLOP to not wait) ###
ob_end_clean(); //just flush all content if exists to request. If server still waiting on answer.
//HERE all script doing in background: Example
$this->db->query('UPDATE new_hook_memory SET new=new+1 WHERE id=1');
EDIT 2019 if you using fastcgi just finish fastcgi and browser close connection but script still will be working up to end.
How finish script: PHP mod_fcgi with fastcgi_finish_request();
For Apache2:
ob_end_flush();
flush();
For php-fpm
fastcgi_finish_request(); $this->db->query('UPDATE new_hook_memory SET new=new+1 WHERE id=1');
Old version:
These two solutions work well for me
( Of course it has been a long time, but I don't think this question is outdated )
using file_get_contents:
//url of php to be called
$url = "example.php/test?id=1";
//this will set the minimum time to wait before proceed to the next line to 1 second
$ctx = stream_context_create(['http'=> ['timeout' => 1]]);
file_get_contents($url,null,$ctx);
//the php will read this after 1 second
using cURL:
//url of php to be called
$url = "example.php/test?id=1";
$test = curl_init();
//this will set the minimum time to wait before proceed to the next line to 100 milliseconds
curl_setopt_array($test,[CURLOPT_URL=>$url,CURLOPT_TIMEOUT_MS=>100,CURLOPT_RETURNTRANSFER=>TRUE]);
curl_exec($test);
//this line will be executed after 100 milliseconds
curl_close ($test);
in both case the called php must set ignore_user_abort(true).
And the result will not be printed in both case, but be careful with the timeout you will set, it needs to be greater than the time that the called php needs to start yielding results.
If possible you can run wget in background (using exec)
There was some frustration in finding a solution that actually works, so I ended up building a service based on fsockopen() that can handle both GET and POST requests, without being blocking.
Below is the service class:
class NonBlockingHttpClientService {
private $method = 'GET';
private $params = [];
private $port = 80;
private $host;
private $path;
private $post_content;
public function isPost(): bool
{
return ($this->method === 'POST');
}
public function setMethodToPost(): NonBlockingHttpClientService
{
$this->method = 'POST';
return $this;
}
public function setPort(int $port): NonBlockingHttpClientService
{
$this->port = $port;
return $this;
}
public function setParams(array $params): NonBlockingHttpClientService
{
$this->params = $params;
return $this;
}
private function handleUrl(string $url): void
{
$url = str_replace(['https://', 'http://'], '', $url);
$url_parts = explode('/', $url);
if(count($url_parts) < 2) {
$this->host = $url_parts[0];
$this->path = '/';
} else {
$this->host = $url_parts[0];
$this->path = str_replace($this->host, '', $url);
}
}
private function handleParams(): void
{
if(empty($this->params)) return;
if($this->isPost()) {
$this->post_content = http_build_query($this->params);
} else {
/*
if you want to specify the params as an array for GET request, they will just be
appended to the path as a query string
*/
if(strpos($this->path, '?') === false) {
$this->path .= '?' . ltrim($this->arrayToQueryString($this->params), '&');
} else {
$this->path .= $this->arrayToQueryString($this->params);
}
}
}
private function arrayToQueryString(array $params): string
{
$string = '';
foreach($params as $name => $value) {
$string .= "&$name=" . urlencode($value);
}
return $string;
}
public function doRequest(string $url): bool
{
$this->handleUrl($url);
$this->handleParams();
$host = $this->host;
$path = $this->path;
$fp = fsockopen($host, $this->port, $errno, $errstr, 1);
if (!$fp) {
$error_message = __CLASS__ . ": cannot open connection to $host$path : $errstr ($errno)";
echo $error_message;
error_log($error_message);
return false;
} else {
fwrite($fp, $this->method . " $path HTTP/1.1\r\n");
fwrite($fp, "Host: $host\r\n");
if($this->isPost()) fwrite($fp, "Content-Type: application/x-www-form-urlencoded\r\n");
if($this->isPost()) fwrite($fp, "Content-Length: " . strlen($this->post_content) . "\r\n");
fwrite($fp, "Connection: close\r\n");
fwrite($fp, "\r\n");
if($this->isPost()) fwrite($fp, $this->post_content);
return true;
}
}
}
It can be used like this:
$req = new NonBlockingHttpClientService();
$req->setMethodToPost(); //default is GET, so just omit this for GET requests
$req->setParams([
'test2' => 'aaaa', //if parameters are specified both with setParams() and in the query string, for GET requests, params specified with setParams() will take precedence
'test3' => 'bbbb',
'time' => date('H:i:s')
]);
$req->doRequest('test.localhost/some_path/slow_api.php?test1=value1&test2=value2');
And the slow_api.php file, can be something like this.
<?php
error_log('start');
sleep(10);
error_log(print_r($_REQUEST, 1) . 'end');
I find it easier to monitor (tail -f) the error log in order to see what is happening.
How can you tell if the request succeeded or not? You need to wait for at least the status code from the server to determine that. If latency is the issue, look at the curl multi API to perform multiple requests in parallel. You should be able to set a write callback function to abort reception of returned data once the status code has been returned.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_TIMEOUT_MS, 1);
curl_exec($ch);
curl_close($ch);
That works well for me.
Tested on PHP 7.1.14 Windows
A bit late now but the solution to this for anyone interested is that CURLOPT_RETURNTRANSFER needs to be set to TRUE, not false. That way the curl_exec function returns a value immediately rather than waiting for the request to complete before returning - i.e. it acts asynchronously rather than synchronously.
Example:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
If you use Linux, you can use shell_exec in your PHP and send the result of curl to dev/null. By this method, you don't receive any answer and after sending PHP goes to the next line and Linux executes your command in the background. You should use this method if the answer is not important to you.
This is an example:
shell_exec("curl 'http://domian.com/message?text=$text' > /dev/null 2>/dev/null &")
Note: You can add any curl option like header or post method or proxy as your need.

Categories