I want to user curl for login to other websites. I use curl multi threading for it.
code 1:
<?php
$ch1 = curl_init();
$ch2 = curl_init();
curl_setopt($ch1, CURLOPT_URL, $url);
curl_setopt($ch2, CURLOPT_URL, $url);
curl_setopt($ch1, CURLOPT_COOKIE, $post);
curl_setopt($ch2, CURLOPT_COOKIE, $post2)
$mh = curl_multi_init();
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active = null;
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
?>
code 2 :
<?php
$mh = curl_multi_init();
foreach($data as $key => $value)
{
$key = curl_init();
curl_setopt($key, CURLOPT_URL, $value["url"]);
curl_setopt($key, CURLOPT_COOKIE, $value[$key]);
curl_multi_add_handle($mh,$key);
}
$active = null;
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
?>
Code 1 can return me result that i want but not very dynamic. Code 2 just return me ‹‹. Anyone can explain to me why like this?
Could you provide an example of $data you are using in the second example. The first loop there makes no sense for me:
foreach($data as $key => $value) //assuming $key here is a scalar
{
$key = curl_init(); // after this line $key is a resource or false
curl_setopt($key, CURLOPT_URL, $value["url"]);
curl_setopt($key, CURLOPT_COOKIE, $value[$key]); // likely $value[$key] is empty, and there is a notice in the log
curl_multi_add_handle($mh,$key);
}
Apart from that, you probably want to close handlers after the final loop, and output some results. I don't see at which point Code 2 just return anything.
Related
At the bottom of the code snippet, I am attempting to remove the curl handle from the multi handle. However PHP reports that it is an invalid curl handle. the curl_close call reports the same thing. I am confused since I have not closed it above that point.
Am i losing it anywhere. I don't see where...
foreach ($urls as $url) {
$request = [];
$request['url'] = $url;
$request['body'] = '';
$request['response_headers'] = [];
$request['curl_handle'] = curl_init();
$url['config'] = json_decode($url['config'], true);
if($url['config']['method'] == 'GET') {
curl_setopt($request['curl_handle'], CURLOPT_HTTPGET, true);
}
curl_setopt($request['curl_handle'], CURLOPT_URL, $url['source_url']);
curl_setopt($request['curl_handle'], CURLOPT_WRITEFUNCTION, function($curl, $body) use (&$request) {
$request['body'] .= $body;
return strlen($body);
});
curl_setopt($request['curl_handle'], CURLOPT_HEADERFUNCTION, function($curl, $header) use (&$request) {
$request['response_headers'][] = $header;
return strlen($header);
});
$followRedirects = boolval($url['config']['follow_redirects']);
curl_setopt($request['curl_handle'], CURLOPT_FOLLOWLOCATION, $followRedirects);
curl_setopt($request['curl_handle'], CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($request['curl_handle'], CURLOPT_TIMEOUT, 120);
curl_setopt($request['curl_handle'], CURLOPT_MAXREDIRS, intval($url['config']['total_redirects']));
curl_setopt($request['curl_handle'], CURLOPT_REDIR_PROTOCOLS, CURLPROTO_HTTP | CURLPROTO_HTTPS);
curl_setopt($request['curl_handle'], CURLOPT_MAXFILESIZE, intval($url['config']['max_download']));
curl_setopt($request['curl_handle'], CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($request['curl_handle'], CURLOPT_SSL_VERIFYPEER, false);
$requests[] = &$request;
}
$mh = curl_multi_init();
//add the handles
foreach ($requests as &$request) {
curl_multi_add_handle($mh, $request['curl_handle']);
}
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
print('after exec');
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
$mrc = curl_multi_exec($mh, $active);
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
print('performing again');
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
if ($mrc != CURLM_OK) {
print("Curl multi read error $mrc\n");
}
foreach ($requests as &$request) {
processResponse($request);
curl_multi_remove_handle($mh, $request['curl_handle']);
curl_close($request['curl_handle']);
}
curl_multi_close($mh);
Probably you need to change following lines:
foreach ($requests as &$request) {
processResponse($request);
curl_multi_remove_handle($mh, $request['curl_handle']);
curl_close($request['curl_handle']);
}
to
foreach ($requests as &$request) {
processResponse($request);
curl_close($request['curl_handle']);
curl_multi_remove_handle($mh, $request['curl_handle']);
}
The issue was that $request was being copied by reference into $requests, so the same curl_handle was being passed through multi curl and then to close_handle multiple times. The fix was not setting $request by reference into $requests.
I changed :
$requests[] = &$request;
to
$requests[] = $request;
That solved my issue.
How can I stop my function from reciving responses when I send cURL requests? I developed this function based on http://php.net/manual/en/function.curl-multi-exec.php documentation. However when I try send over 50 messages the app I am developing freezes at the send button for over 20 seconds. Is there a way to send multiple cURL and force the code to not wait for a response?
function send_multi_data($url,$data){
//create the multiple cURL handle
$mh = curl_multi_init();
$active = null;
$count=0;
// set URL and other appropriate options
foreach($data as $message){
$count++;
$ch[$count] = curl_init();
curl_setopt($ch[$count], CURLOPT_URL, $url);
curl_setopt($ch[$count], CURLOPT_HEADER, 0);
curl_setopt ($ch[$count], CURLOPT_POSTFIELDS, $message);
curl_multi_add_handle($mh,$ch[$count]);
}
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
// Close the handle
$count1=0;
foreach($data as $message){
$count1++;
curl_multi_remove_handle($mh, $ch[$count1]);
}
curl_multi_close($mh);
}
I created a php script to run some google queries in order to get acquainted with the concept of multiple parallel requests in curl. As a basis, I used example #1 on this page: http://php.net/manual/en/function.curl-multi-exec.php
I found that curl_multi_select in the provided example always returns -1. The documentation states that this indicates some error happened (by the underlying system call), but there seems to be no way to deduce what went wrong.
Code
$queries = array("Mr.", "must", "my", "name", "never", "new", "next", "no", "not", "now");
$mh = curl_multi_init();
$handles = array();
foreach($queries as $q)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.google.nl/#q=" . $q);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_multi_add_handle($mh,$ch);
$handles[] = $ch;
}
echo "created\n";
$active = null;
$mrc = curl_multi_exec($mh, $active);
if ($mrc != CURLM_OK)
throw new Exception("curl_multi_exec failed");
while ($active && $mrc == CURLM_OK) {
$select = curl_multi_select($mh);
if ($select != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
} else throw new Exception("curl_multi_select failed (it returned -1)");
}
// removed cleanup code for briefety.
Question
How can I find out why curl_multi_select returns -1 or why does it return -1? Do I need some special configuration of php to allow for this multithreading?
As a comment says at http://php.net/manual/en/function.curl-multi-init.php#115055, there's a problem on the official document.
I don't know why and don't have enough knowledge on libcurl, but I understand that curl_multi_select($mh) always have a chance to return -1;
So, this snippets (from the above url) works for me.
<?php
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) == -1) {
usleep(100);
}
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
?>
I am getting only the result for the first script. But I want the response for both PHP scripts to be in one output. In my program I am developing, I am querying several remote servers. I have to search information on different servers and return the result in one response.
<?php
// build the individual requests as above, but do not execute them
$ch_1 = curl_init('http://lifesaver.netai.net/example/pharm.php');
$ch_2 = curl_init('http://192.168.1.2/example/pharm.php');
curl_setopt($ch_1, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch_2, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch_1, CURLOPT_POST,1);
curl_setopt($ch_2, CURLOPT_POST,1);
curl_setopt($ch_1, CURLOPT_POSTFIELDS, "name=$value");
curl_setopt($ch_2, CURLOPT_POSTFIELDS, "name=$value");
// $_POST["name"]
// build the multi-curl handle, adding both $ch
$mh = curl_multi_init();
curl_multi_add_handle($mh, $ch_1);
curl_multi_add_handle($mh, $ch_2);
// execute all queries simultaneously, and continue when all are complete
$running = null;
do {
curl_multi_exec($mh, $running);
} while ($running);
// all of our requests are done, we can now access the results
$response_1 = curl_multi_getcontent($ch_1);
$response_2 = curl_multi_getcontent($ch_2);
echo $response_1;
echo $response_2;
?>
According to the documentation of curl_multi_getcontent, you need to set CURLOPT_RETURNTRANSFER to true instead of false.
Also, the documentation of curl_multi_init shows an example which is slightly different than yours when it comes to the execution loop:
//execute the handles
$running = null;
do {
$mrc = curl_multi_exec($mh, $running);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($running && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $running);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
You could try it that way.
I'm trying to run this simple piece of code from php.net under my Mac OS X (Mavericks) to try cURL multi exec feature :
<?php
// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "http://lxr.php.net/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
//close the handles
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
?>
That does not work as the only output I got is :
PHP Fatal error: Maximum execution time of 30 seconds exceeded in
This is my environment: Mac OS X 10.9, PHP 5.4.17, Apache 2.2.24.
cUrl is installed as I my regular curl "single" requests work great.
I think this is an issue with Mac OS but I can't find any fix. Do you have any idea?
EDIT: I tried the same code on a Linux server and everything worked fine.
On php 5.3.18+ be aware that curl_multi_select() may return -1 forever until you call curl_multi_exec().
Try this:
while ($this->active && $mrc == CURLM_OK)
{
// add this line
while (curl_multi_exec($this->mh, $this->active) === CURLM_CALL_MULTI_PERFORM);
if (curl_multi_select($this->mh) != -1)
{
do {
$mrc = curl_multi_exec($this->mh, $this->active);
if ($mrc == CURLM_OK)
{
while($info = curl_multi_info_read($this->mh))
{
$this->process($info);
}
}
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
See https://bugs.php.net/bug.php?id=63411 or http://marchtea.com/?p=109 for more information.