I am fetching the data using the curl in php but as many data are present it returning the 0 output. I am providing my code below.
$result = array();
// multi handle
$mh = curl_multi_init();
$idArr=[2,147,92];
foreach ($idArr as $key => $value) {
$fetchURL = "http://example.com/index.php/rest/V1/categories/".$value."/products/";
//echo $fetchURL.'<br>';
$multiCurl[$key] = curl_init();
curl_setopt($multiCurl[$key], CURLOPT_URL,$fetch_url);
curl_setopt($multiCurl[$key], CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($multiCurl[$key], CURLOPT_HEADER,array("Content-Type: application/json", "Authorization: Bearer " . json_decode($token)));
curl_setopt($multiCurl[$key], CURLOPT_RETURNTRANSFER,true);
curl_multi_add_handle($mh, $multiCurl[$key]);
}
$index=null;
do {
curl_multi_exec($mh,$index);
} while($index > 0);
// get content and remove handles
foreach($multiCurl as $k => $ch) {
$result[$k] = curl_multi_getcontent($ch);
curl_multi_remove_handle($mh, $ch);
}
// close
curl_multi_close($mh);
print_r($result);
Here I have to pass the multiple request and get the result but in this case no result is coming. While I am using simple curl the result is coming. Here my requirement is to reduce the response time.
You are missing the implementation of "curl_multi_select". I have done it for you but not tested. Give it a go
$result = array();
// multi handle
$mh = curl_multi_init();
$multiCurl = array();
$idArr=[2,147,92];
foreach ($idArr as $key => $value) {
$fetchURL = "http://example.com/index.php/rest/V1/categories/".$value."/products/";
//echo $fetchURL.'<br>';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$fetch_url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_HEADER,array("Content-Type: application/json", "Authorization: Bearer " . json_decode($token)));
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
$multiCurl[$key] = $ch;
curl_multi_add_handle($mh, $ch);
}
$active =null;
do {
$mrc = curl_multi_exec($mh,$active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
foreach($multiCurl as $k => $ch) {
$result[$k] = curl_multi_getcontent($ch);
curl_multi_remove_handle($mh, $ch);
}
// close
curl_multi_close($mh);
print_r($result);
I think you can't get response due to set wrong variable in curl option set.Your third line in foreach loop should be like that
curl_setopt($multiCurl[$key], CURLOPT_URL,$fetchURL);
also you write wrong syntex for headers option replace CURLOPT_HEADER by CURLOPT_HTTPHEADER
here is sample code that works perfectly
$idArr=[20,18,21];
$mh = curl_multi_init();
$requests = array();
$curl_objs_arr = [];
foreach ($idArr as $key => $cat) {
$fetchURL = "http:example.com/v2/products?category=".$cat;
$requests[$key] = curl_init($fetchURL);
curl_setopt($requests[$key], CURLOPT_URL,$fetchURL);
curl_setopt($requests[$key], CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($requests[$key], CURLOPT_HTTPHEADER,array("Content-Type: application/json","Authorization: Bearer " . json_decode($token)));
curl_setopt($requests[$key], CURLOPT_RETURNTRANSFER,true);
curl_multi_add_handle($mh, $requests[$key]);
}
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);
foreach ($requests as $key => $request) {
$result[$key] = curl_multi_getcontent($request);
curl_multi_remove_handle($mh, $request);
}
curl_multi_close($mh);
echo "<pre>";
print_r($result);exit;
set your code by this way, that's solve your problem.
Related
I have a function which uses curl_multi to make several GET requests. However, I can't figure out how to turn the response to JSON so I could access it on the front end. Usually I use json_decode(), however, it is not working this time. Maybe it's because I'm trying to decode a whole array of strings instead of a singular string?. Currently the response looks like this on the front-end:
And here's my function:
public function getVisited($username){
$user = User::where('username', $username)->first();
$visitedPlaces = $user->visitedPlaces;
$finalPlaces = [];
foreach ($visitedPlaces as $visitedPlace) {
$url = "https://maps.googleapis.com/maps/api/place/details/json?placeid=" . $visitedPlace->place_id . "&key=AIzaSyDQ64lYvtaYYYNWxLzkppdN-n0LulMOf4Y";
array_push($finalPlaces, $url);
}
$result = array();
$curly = array();
$mh = curl_multi_init();
foreach ($finalPlaces as $id => $d) {
$curly[$id] = curl_init();
$url = $d;
curl_setopt($curly[$id], CURLOPT_URL, $url);
curl_setopt($curly[$id], CURLOPT_HEADER, 0);
curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curly[$id], CURLOPT_SSL_VERIFYPEER, 0); // Skip SSL Verification
curl_multi_add_handle($mh, $curly[$id]);
}
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);
foreach($curly as $id => $c) {
$result[$id] = curl_multi_getcontent($c);
curl_multi_remove_handle($mh, $c);
}
curl_multi_close($mh);
return response()->json([
'sights' => $result
], 201);
}
If you are expecting a json string from the curl_multi_getcontent($c) then you should decode it:
$result[$id] = json_decode(curl_multi_getcontent($c));
Then it should be encoded in the response appropriately.
At the bottom of the code snippet, I am attempting to remove the curl handle from the multi handle. However PHP reports that it is an invalid curl handle. the curl_close call reports the same thing. I am confused since I have not closed it above that point.
Am i losing it anywhere. I don't see where...
foreach ($urls as $url) {
$request = [];
$request['url'] = $url;
$request['body'] = '';
$request['response_headers'] = [];
$request['curl_handle'] = curl_init();
$url['config'] = json_decode($url['config'], true);
if($url['config']['method'] == 'GET') {
curl_setopt($request['curl_handle'], CURLOPT_HTTPGET, true);
}
curl_setopt($request['curl_handle'], CURLOPT_URL, $url['source_url']);
curl_setopt($request['curl_handle'], CURLOPT_WRITEFUNCTION, function($curl, $body) use (&$request) {
$request['body'] .= $body;
return strlen($body);
});
curl_setopt($request['curl_handle'], CURLOPT_HEADERFUNCTION, function($curl, $header) use (&$request) {
$request['response_headers'][] = $header;
return strlen($header);
});
$followRedirects = boolval($url['config']['follow_redirects']);
curl_setopt($request['curl_handle'], CURLOPT_FOLLOWLOCATION, $followRedirects);
curl_setopt($request['curl_handle'], CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($request['curl_handle'], CURLOPT_TIMEOUT, 120);
curl_setopt($request['curl_handle'], CURLOPT_MAXREDIRS, intval($url['config']['total_redirects']));
curl_setopt($request['curl_handle'], CURLOPT_REDIR_PROTOCOLS, CURLPROTO_HTTP | CURLPROTO_HTTPS);
curl_setopt($request['curl_handle'], CURLOPT_MAXFILESIZE, intval($url['config']['max_download']));
curl_setopt($request['curl_handle'], CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($request['curl_handle'], CURLOPT_SSL_VERIFYPEER, false);
$requests[] = &$request;
}
$mh = curl_multi_init();
//add the handles
foreach ($requests as &$request) {
curl_multi_add_handle($mh, $request['curl_handle']);
}
$active = null;
//execute the handles
do {
$mrc = curl_multi_exec($mh, $active);
print('after exec');
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
$mrc = curl_multi_exec($mh, $active);
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
print('performing again');
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
if ($mrc != CURLM_OK) {
print("Curl multi read error $mrc\n");
}
foreach ($requests as &$request) {
processResponse($request);
curl_multi_remove_handle($mh, $request['curl_handle']);
curl_close($request['curl_handle']);
}
curl_multi_close($mh);
Probably you need to change following lines:
foreach ($requests as &$request) {
processResponse($request);
curl_multi_remove_handle($mh, $request['curl_handle']);
curl_close($request['curl_handle']);
}
to
foreach ($requests as &$request) {
processResponse($request);
curl_close($request['curl_handle']);
curl_multi_remove_handle($mh, $request['curl_handle']);
}
The issue was that $request was being copied by reference into $requests, so the same curl_handle was being passed through multi curl and then to close_handle multiple times. The fix was not setting $request by reference into $requests.
I changed :
$requests[] = &$request;
to
$requests[] = $request;
That solved my issue.
I have the following function, running an array of URLs with cURL:
function multiRequest($urls){
$res = array();
// Create get requests for each URL
$mh = curl_multi_init();
foreach($urls as $i => $url)
{
$ch[$i] = curl_init($url);
curl_setopt($ch[$i], CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch[$i], CURLOPT_HEADER, "Accept: application/json");
curl_setopt($ch[$i], CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt ($ch[$i], CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt ($ch[$i], CURLOPT_SSL_VERIFYPEER, 0);
curl_multi_add_handle($mh, $ch[$i]);
}
// Start performing the request
do {
$execReturnValue = curl_multi_exec($mh, $runningHandles);
} while ($execReturnValue == CURLM_CALL_MULTI_PERFORM);
// Loop and continue processing the request
while ($runningHandles && $execReturnValue == CURLM_OK) {
// Wait forever for network
$numberReady = curl_multi_select($mh);
if ($numberReady != -1) {
// Pull in any new data, or at least handle timeouts
do {
$execReturnValue = curl_multi_exec($mh, $runningHandles);
} while ($execReturnValue == CURLM_CALL_MULTI_PERFORM);
}
}
// Check for any errors
if ($execReturnValue != CURLM_OK) {
trigger_error("Curl multi read error $execReturnValue\n", E_USER_WARNING);
}
// Extract the content
foreach($urls as $i => $url)
{
// Check for errors
$curlError = curl_error($ch[$i]);
if($curlError == "") {
$res[$i] = curl_multi_getcontent($ch[$i]);
} else {
print "Curl error on handle $i: $curlError\n";
}
// Remove and close the handle
curl_multi_remove_handle($mh, $ch[$i]);
curl_close($ch[$i]);
}
// Clean up the curl_multi handle
curl_multi_close($mh);
// Print the response data
return $res;
}
I get this error: Maximum execution time of 30 seconds exceeded, the line it points to is $numberReady = curl_multi_select($mh);
The code used to work on PHP version 5.3, after updating to 5.5 the code is no longer working.
I tried setting the limit to more than 30 seconds, it didn't help, also tried ini_set('max_execution_time', 0);
Is there anything wrong with my code? Is there another way to debug it?
This alternative function solved it for me:
function multiRequest($nodes){
//function multiple_threads_request($nodes){
$mh = curl_multi_init();
$curl_array = array();
foreach($nodes as $i => $url) {
$curl_array[$i] = curl_init($url);
curl_setopt($curl_array[$i], CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_array[$i], CURLOPT_HEADER, "Accept: application/json");
curl_setopt($curl_array[$i], CURLOPT_USERPWD, xxxx);
curl_multi_add_handle($mh, $curl_array[$i]);
}
$running = NULL;
do {
usleep(10000);
curl_multi_exec($mh,$running);
} while($running > 0);
$res = array();
foreach($nodes as $i => $url) {
$res[$url] = curl_multi_getcontent($curl_array[$i]);
}
foreach($nodes as $i => $url) {
curl_multi_remove_handle($mh, $curl_array[$i]);
}
curl_multi_close($mh);
return $res;
}
EDITED
I am attempting to use curl multi to check the response from a website and additionally check each curl response for a portion of text. I have grouped the data into an array but I cannot figure out if I am using the correct/most efficient method to run the strpos() function using the 'post' text.
$data = array(array());
$data[0]['url'] = 'http://www.google.com';
$data[0]['post'] = 'google text';
$data[1]['url'] = 'http://www.yahoo.com';
$data[1]['post'] = 'yahoo text';
$r = multiRequest($data);
echo '<pre>';
print_r($r);
Here is my function:
function multiRequest($data, $options = array()) {
// array of curl handles
$curly = array();
// data to be returned
$result = array();
// multi handle
$mh = curl_multi_init();
// loop through $data and create curl handles
// then add them to the multi-handle
foreach ($data as $id => $d) {
$curly[$id] = curl_init();
$url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
curl_setopt($curly[$id], CURLOPT_URL, $url);
curl_setopt($curly[$id], CURLOPT_HEADER, 0);
curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curly[$id], CURLOPT_USERAGENT, 'Chrome: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2');
// extra options?
if (!empty($options)) {
curl_setopt_array($curly[$id], $options);
}
curl_multi_add_handle($mh, $curly[$id]);
}
// execute the handles
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);
// get content and remove handles
foreach($curly as $id => $c) {
$result[$id][] = curl_getinfo($c, CURLINFO_EFFECTIVE_URL);
$result[$id][] = curl_getinfo($c, CURLINFO_HTTP_CODE);
$result[$id][] = curl_getinfo($c, CURLINFO_CONTENT_TYPE);
$url = curl_getinfo($c, CURLINFO_EFFECTIVE_URL);
// loop data again
foreach ($data as $id => $d){
if($url==$d['url']){ // only check current url data
$text = curl_exec($c);
$result[$id][] = strpos($text, $d['post']);
}
}
curl_multi_remove_handle($mh, $c);
}
// all done
curl_multi_close($mh);
return $result;
}
Can anyone advise on whether my solution is appropriate? Is there a more efficient/better way to perform my strpos() check?
Thanks
Use an array so you can relate all the urls, strings, and curl handles together:
$stuff = array(
0 => array('url' => 'google', 'text' => 'googletext', 'curl' => null)
1 => array('url' => 'yahoo', 'text' => 'yahootext', 'curl' => null)
etc..
);
foreach($stuff as $key => $info) {
$stuff[$key]['curl'] = curl_init($stuff[$key]['url']);
curl_multi_add_handle($mh, $stuff[$key]['curl']);
}
Then do a similar loop when you're processing the results.
I'm having trouble creating multiple xml requests using php's curl_multi_exec.
The problem is that the do...while loop containing the curl_multi_exec command runs only once and then quits.
Resources Used:
http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/
http://php.net/manual/en/function.curl-multi-exec.php/
http://www.rustyrazorblade.com/2008/02/curl_multi_exec/
Take a look at my code:
//Multi handle curl initialization
$mh = curl_multi_init();
//set url
$url = 'my_url';
foreach($latLng as $id => $l) {
$ch[$id] = curl_init();
//$request previously set
//Initialize and set options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $request);
//add to multi_handle
curl_multi_add_handle($mh, $ch[$id]);
}
//Execute the handles
$running = null;
do {
$mrc = curl_multi_exec($mh, $running);
$ready=curl_multi_select($mh);
echo "Ran once\n";
} while ($mrc == CURLM_CALL_MULTI_PERFORM && $ready > 0);
while ($active && $mrc == CURLM_OK) {
if ($curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $running);
echo "Ran again\n";
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
foreach ($mh as $c) {
// HTTP response code
$code = curl_getinfo($c, CURLINFO_HTTP_CODE);
// cURL error number
$curl_errno = curl_errno($c);
// cURL error message
$curl_error = curl_error($c);
// output if there was an error
if ($curl_error) {
echo("*** cURL error: ($curl_errno) $curl_error\n");
}
}
//get content and remove handles
foreach ($ch as $c) {
$result[] = curl_multi_getcontent($c);
curl_multi_remove_handle($mh, $c);
}
print_r($result);
//Close curl
curl_multi_close($mh);
}
I know the request is valid because I receive the correct return data when I perform a single curl execution. The problem lies with the curl_multi_exec().
The output I am receiving is "Ran once" followed by the empty arrays of the curl_multi_getcontent() calls. See below:
Ran once
Array
(
[0] =>
[1] =>
[2] =>
[3] =>
[4] =>
[5] =>
[6] =>
[7] =>
[8] =>
)
Any help is greatly appreciated.
You're not setting up the curl options correctly:
Currently, you're setting options on $ch which is your array, you need to be setting the options specifically on the current curl handler, which in your loop is $ch[$id]:
//Initialize and set options
curl_setopt($ch[$id], CURLOPT_URL, $url);
curl_setopt($ch[$id], CURLOPT_HEADER, 0);
curl_setopt($ch[$id], CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch[$id], CURLOPT_POST, 1);
curl_setopt($ch[$id], CURLOPT_POSTFIELDS, $request);
change this:
foreach ($mh as $c) {
$code = curl_getinfo($c, CURLINFO_HTTP_CODE);
to:
for($i=1;$i<=count($array);$i++){
$code = curl_multi_getcontent($ch[$i]);
assuming $array is the array for your multiple $url.