From the PHP manual a multi curl is performed like this:
// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "http://lxr.php.net/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
//---------
Link:http://php.net
I can copy paste the code as is and it would get me the content. Notice how I don't "echo" anything, so it works without echoing.
So my question is, where does the data come from? What is the object holding the data? I know you have to set CURLOPT_RETURNTRANSFER to true and then get the content with curl_multi_getcontent() but as I stated the script retrieves content, but where is the object?
Both curl_exec and curl_multi_exec output the response by default. You need to set CURLOPT_RETURNTRANSFER option to true to disable the output and make those functions return the response instead.
Related
In the code below from http://php.net/manual/en/function.curl-multi-init.php
How can I add code before the second request is made (ie sleep(5)) before curl makes the request to twitter)
Regards
<?php
// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "https://www.google.com");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "https://twitter.com");
curl_setopt($ch2, CURLOPT_HEADER, 0);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
//close the handles
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
?>
I'm no PHP guy or competent programmer for that matter :D Now that disclaimer is out there, here's my solution.
There's probably a much cleaner way to do this but I have limited knowledge of PHP and how to extend classes. For that reason, I decided to use the built-in process control extensions and create a helper function to handle the curl process. I'm sure there are much better programmers out there ready to provide a much cleaner solution though.
<?php
// Helper function
function async_curl($url,$delay){
sleep($delay);
echo "FORK: Getting $url after $delay seconds\n";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
// Mute the return for demonstration purposes.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_exec($ch);
curl_close($ch);
}
$urls = array("http://google.com","http://twitter.com","http://www.facebook.com");
foreach($urls as $url){
// Generate random timeout for demonstration purposes.
$delay = rand(1,20);
// Create a forked child process for each URL
$pid = pcntl_fork();
// Exit if fork failed
if ($pid == -1) {
exit("Error, failed to create a child process for the URL: $url");
// Create a single child process to call the helper function
} else if ($pid == 0) {
echo "MAIN: Forking process for $url\nPID: " .getmypid() . "\tDelay: $delay\n";
async_curl($url,$delay);
exit();
}
}
// Wait for all forked processes to complete before exiting.
while (($pid = pcntl_waitpid(0, $status)) > 0) {
echo "MAIN: Process $pid completed\n";
}
?>
I created a php script to run some google queries in order to get acquainted with the concept of multiple parallel requests in curl. As a basis, I used example #1 on this page: http://php.net/manual/en/function.curl-multi-exec.php
I found that curl_multi_select in the provided example always returns -1. The documentation states that this indicates some error happened (by the underlying system call), but there seems to be no way to deduce what went wrong.
Code
$queries = array("Mr.", "must", "my", "name", "never", "new", "next", "no", "not", "now");
$mh = curl_multi_init();
$handles = array();
foreach($queries as $q)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.google.nl/#q=" . $q);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_multi_add_handle($mh,$ch);
$handles[] = $ch;
}
echo "created\n";
$active = null;
$mrc = curl_multi_exec($mh, $active);
if ($mrc != CURLM_OK)
throw new Exception("curl_multi_exec failed");
while ($active && $mrc == CURLM_OK) {
$select = curl_multi_select($mh);
if ($select != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
} else throw new Exception("curl_multi_select failed (it returned -1)");
}
// removed cleanup code for briefety.
Question
How can I find out why curl_multi_select returns -1 or why does it return -1? Do I need some special configuration of php to allow for this multithreading?
As a comment says at http://php.net/manual/en/function.curl-multi-init.php#115055, there's a problem on the official document.
I don't know why and don't have enough knowledge on libcurl, but I understand that curl_multi_select($mh) always have a chance to return -1;
So, this snippets (from the above url) works for me.
<?php
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) == -1) {
usleep(100);
}
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
?>
I am getting only the result for the first script. But I want the response for both PHP scripts to be in one output. In my program I am developing, I am querying several remote servers. I have to search information on different servers and return the result in one response.
<?php
// build the individual requests as above, but do not execute them
$ch_1 = curl_init('http://lifesaver.netai.net/example/pharm.php');
$ch_2 = curl_init('http://192.168.1.2/example/pharm.php');
curl_setopt($ch_1, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch_2, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch_1, CURLOPT_POST,1);
curl_setopt($ch_2, CURLOPT_POST,1);
curl_setopt($ch_1, CURLOPT_POSTFIELDS, "name=$value");
curl_setopt($ch_2, CURLOPT_POSTFIELDS, "name=$value");
// $_POST["name"]
// build the multi-curl handle, adding both $ch
$mh = curl_multi_init();
curl_multi_add_handle($mh, $ch_1);
curl_multi_add_handle($mh, $ch_2);
// execute all queries simultaneously, and continue when all are complete
$running = null;
do {
curl_multi_exec($mh, $running);
} while ($running);
// all of our requests are done, we can now access the results
$response_1 = curl_multi_getcontent($ch_1);
$response_2 = curl_multi_getcontent($ch_2);
echo $response_1;
echo $response_2;
?>
According to the documentation of curl_multi_getcontent, you need to set CURLOPT_RETURNTRANSFER to true instead of false.
Also, the documentation of curl_multi_init shows an example which is slightly different than yours when it comes to the execution loop:
//execute the handles
$running = null;
do {
$mrc = curl_multi_exec($mh, $running);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($running && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $running);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
You could try it that way.
I'm looking to optimize my application. It uses the Twitter and the Facebook API and loads large files to be displayed on the users screen. Right now, I am running the script linearly, calling one file that includes both API calls using AJAX and loading all of the information onto the screen. Would it be faster for me to separate the two API calls into two different files and then load each one separately with AJAX? This way, if one response was taking longer then the other, the faster one would still be displayed.
Thank you.
If it matters, I'm using PHP and CURL for API calls.
Certainly it would be better if the AJAX calls don't depend each other. You can also do this at the PHP side using curl_multi_init that executes HTTP calls in paralell.
Sample from the PHP manual:
<?php
// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "http://lxr.php.net/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
//close the handles
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
?>
i want to get several pages thru curl_exec, first page is come normally, but all others - 302 header, what reason?
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, ROOT_URL);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$content = curl_exec($curl); // here good content
curl_close($curl);
preg_match_all('/href="(\/users\/[^"]+)"[^>]+>\s*/i', $content, $p);
for ($j=0; $j<count($p[1]); $j++){
$new_curl = curl_init();
curl_setopt($new_curl, CURLOPT_URL, NEW_URL.$p[1][$j]);
curl_setopt($new_curl, CURLOPT_RETURNTRANSFER, 0);
$content = curl_exec($new_curl); // here 302
curl_close($new_curl);
preg_match('/[^#]+#[^"]+/i', $content, $p2);
}
smth like this
You probably want to provide a sample of your code so we can see if you're omitting something.
302 response code typically indicates that the server is redirecting you to a different location (found in the Location response header). Depending on what flags you use, CURL can either retrieve that automatically or you can watch for the 302 response and retrieve it yourself.
Here is how you would get CURL to follow the redirects (where $ch is the handle to your curl connection):
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);// allow redirects
You can use curl multi which is faster and can get data from all the url's in parallel.
You can use it like this
//Initialize
$curlOptions = array(CURLOPT_RETURNTRANSFER => 1);//Add whatever u additionally want.
$curlHandl1 = curl_init($url1);
curl_setopt_array($curlHandl1, $curlOptions);
$curlHandl2 = curl_init($url2);
curl_setopt_array($curlHandl2, $curlOptions);
$multi = curl_multi_init();
curl_multi_add_handle($multi, $curlHandle1);
curl_multi_add_handle($multi, $curlHandle2);
//Run Handles
$running = null;
do {
$status = curl_multi_exec($mh, $running);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($running && $status == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$status = curl_multi_exec($mh, $running);
} while ($status == CURLM_CALL_MULTI_PERFORM);
}
}
//Retrieve Results
$response1 = curl_multi_getcontent($curlHandle1);
$status1 = curl_getinfo($curlHandle1);
$response1 = curl_multi_getcontent($curlHandle1);
$status1 = curl_getinfo($curlHandle1);
You can find more information here http://www.php.net/manual/en/function.curl-multi-exec.php
Checkout the Example1