Multiple cURL requests are better to be made in an asynchronous manner, that is without each of the requests waiting till all the previous requests have received responses. Another optimization in many cases would be starting to process a received response without waiting for other responses. However, the docs and official examples are not clear when it is both possible and as early as possible to check for completed requests (which is typically done using curl_multi_info_read function).
So when is the earliest point to check for completed requests? Or what is the optimal set of such points?
This is the example from the curl_multi_exec's page (comments in upper case are mine):
<?php
// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "http://lxr.php.net/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
// SHOULD REQUESTS BE CHECKED FOR COMPLETION HERE?
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
// SHOULD REQUESTS BE CHECKED FOR COMPLETION HERE?
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
// SHOULD REQUESTS BE CHECKED FOR COMPLETION HERE?
}
// SHOULD REQUESTS BE CHECKED FOR COMPLETION HERE?
}
// SHOULD REQUESTS BE CHECKED FOR COMPLETION HERE?
//close the handles
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
?>
First, to simplify your life the CURLM_CALL_MULTI_PERFORM return code isn't used in modern libcurls (not used in 7.20.0 or later).
Then, as long as 'active' is larger than zero there are at least one active transfer in progress so you can wait with checking curl_multi_info_read() if you want.
Or you can call curl_multi_info_read() immediately after every call to curl_multi_exec(), that's up to you!
Related
I have a little question:
public function backend()
{
$form = $_POST;
// execute this code in another thread (don't wait to finish)
SearchOnGoogle($form);
// redirect instantly (and function "SearchOnGoogle" works in background);
return redirect('/');
}
How can I do this thing? I tried a lot of things, and the function redirect is executed when SearchOnGoogle finishies execution.
If you want to do this within PHP then you can use curl_multi-exec(). This does does not execute "in the background" - but most of the time a client is retrieving content over HTTP, it is simply waiting for data to pass across the network. The curl_multi_select() function deals with checking the task(s) to see if if the local machine needs to do anything or if any data has arrived from the remote system.
The documentation for curl_multi_exec() in the PHP manual is (IMHO) not on a par with the writeups for other functions. There is a deeper dive here, briefly....
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "http://example.com/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
$requests=array($ch1, $ch2);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active=count($requests);
$started=microtime(true);
for ($x=0; $x<=4 && $active; $x++) {
curl_multi_exec($mh, $active);
// we wait for a bit to allow stuff TCP handshakes to complete and so forth...
curl_mutli_select($mh, 0.02)
}
do_something_useful();
do {
// wait for everything to finish...
curl_multi_exec($mh, $active);
if ($active) {
curl_mutli_select($mh, 0.05);
use_some_spare_cpu_cuycles_here();
}
// until all the results are in or a timeout occurs
} while ($active > 0 && (MAX_RUNTIME<microtime(true)=$started);
I have an API written in PHP that sends 10 requests with CURL.
The problem is that when I send a HTTP request to the API, I get the response right away, although the server hasn't finished working( getting the response for all of the 10 requests).
I can't use ignore_user_abort() because I need to know exactly the time that the API finished.
How can I notify the connection "hey, wait for the script to finish working"?
Important note: if I use sleep() the connection holds.
Here's my code: gist
This is just a example to show how ob_start works.
echo "hello";
ob_start(); // output buffering starts here
echo "hello1";
//all curl requests
if(all curl requests completed)
{
ob_end_flush() ;
}
With no code to refer, I can only show implementation of ob_start. You have to change this code according to your requirement.
$handlers = [];
$mh = curl_multi_init();
ob_start(); // output buffering starts here
foreach($query->fetchAll() as $domain){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://'.$domain['name']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $DEFAULT_REQUEST_TIMEOUT);
curl_setopt($ch, CURLOPT_TIMEOUT, $DEFAULT_REQUEST_TIMEOUT);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 2);
curl_multi_add_handle($mh, $ch);
$handlers[] = ['ch'=>$ch, 'domain_id'=>$domain['domain_id']];
echo $domain['name'];
}
// Execute the handles
$active = null;
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
// Wait for activity on any curl-connection
if (curl_multi_select($mh) == -1) {
usleep(1);
}
// Continue to exec until curl is ready to
// give us more data
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
// Extract the content
$values = [];
foreach($handlers as $key => $handle){
// Check for errors
echo $key.'. result: ';
$curlError = curl_error($handle['ch']);
if($curlError == ""){
$res = curl_multi_getcontent($handle['ch']);
echo 'done';
}
else {
echo "Curl error on handle $key: $curlError".' <br />';
}
// Remove and close the handle
curl_multi_remove_handle($mh, $handle['ch']);
curl_close($handle['ch']);
}
// Clean up the curl_multi handle
curl_multi_close($mh);
ob_end_flush() ; // output flushed here
Source - http://php.net/manual/en/function.ob-start.php
I use this code for my website
ob_start("unique_identifier");
// your header script
// your page script
// your footer script
ob_end_flush("unique_identifier");
ob_end_clean("unique_identifier");
I use "unique_identifier" because inside my script also exists another
ob_start()
I'm looking to optimize my application. It uses the Twitter and the Facebook API and loads large files to be displayed on the users screen. Right now, I am running the script linearly, calling one file that includes both API calls using AJAX and loading all of the information onto the screen. Would it be faster for me to separate the two API calls into two different files and then load each one separately with AJAX? This way, if one response was taking longer then the other, the faster one would still be displayed.
Thank you.
If it matters, I'm using PHP and CURL for API calls.
Certainly it would be better if the AJAX calls don't depend each other. You can also do this at the PHP side using curl_multi_init that executes HTTP calls in paralell.
Sample from the PHP manual:
<?php
// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "http://lxr.php.net/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
//close the handles
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
?>
I am using a php script to upload lot of files. I am using the CURL command . The remote server accepts only POST requests. But when I execute the below script it processes the first request and waits until the first file is uploaded. Is there a way to make it non blocking and run simultaneous 2 curl upload requests .Find the code sample below.
<?php
$arr= array(somefile1.txt,somefile2.txt);
for ( $i=0;$i<2;$i++) {
$cmd = "curl -F name=aaa -F type=yyy FileName=#/xxxxx/xxxx/$arr[$i] http://someurl.com";
print "Executing file ";
shell_exec("nohup $cmd 2> /dev/null & echo $!" );
print "======= done ================";
}
?>
I believe you may want curl_multi_init. Here is an outbound example; it will have to be adapted for your inbound problem. This seems cleaner than you forking multiple threads yourself.
<?php
// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "http://lxr.php.net/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
//close the handles
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
?>
There is a good article about "multithreading", take a look at it here: Multithreading in PHP with CURL
You can try use PHP Simple Curl Wrapper - https://github.com/Graceas/php-simple-curl-wrapper. This library allows the processing of multiple request's asynchronously.
You can find full answer here: php asynchronous cURL request
No you cannot run simultaneously two curl statements.
Curl is made for working like this. A Curl statement will make the later statements
wait until it finishes its operation.
I found this script on php.net and lets say I wanted to get only the info from part of the page. How would one go about doing this, I know how to do it with curl_init, but the multi seems much more efficent.
For example:
from php.net
<?php
// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "http://lxr.php.net/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
//close the handles
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
?>
Id like to just get the info below from the request:
<b>Key enhancements in PHP 5.3.3 include:</b>
</p>
<ul>
<li>Upgraded bundled sqlite to version 3.6.23.1.</li>
<li>Upgraded bundled PCRE to version 8.02.</li>
<li>Added FastCGI Process Manager (FPM) SAPI.</li>
<li>Added stream filter support to mcrypt extension.</li>
<li>Added full_special_chars filter to ext/filter.</li>
<li>Fixed a possible crash because of recursive GC invocation.</li>
<li>Fixed bug #52238 (Crash when an Exception occured in iterator_to_array).</li>
<li>Fixed bug #52041 (Memory leak when writing on uninitialized variable returned from function).</li>
<li>Fixed bug #52060 (Memory leak when passing a closure to method_exists()).</li>
<li>Fixed bug #52001 (Memory allocation problems after using variable variables).</li>
<li>Fixed bug #51723 (Content-length header is limited to 32bit integer with Apache2 on Windows).</li>
<li>Fixed bug #48930 (__COMPILER_HALT_OFFSET__ incorrect in PHP >= 5.3).</li>
</ul>