Combining curl multi and strpos() to find text on webpage - php

EDITED
I am attempting to use curl multi to check the response from a website and additionally check each curl response for a portion of text. I have grouped the data into an array but I cannot figure out if I am using the correct/most efficient method to run the strpos() function using the 'post' text.
$data = array(array());
$data[0]['url'] = 'http://www.google.com';
$data[0]['post'] = 'google text';
$data[1]['url'] = 'http://www.yahoo.com';
$data[1]['post'] = 'yahoo text';
$r = multiRequest($data);
echo '<pre>';
print_r($r);
Here is my function:
function multiRequest($data, $options = array()) {
// array of curl handles
$curly = array();
// data to be returned
$result = array();
// multi handle
$mh = curl_multi_init();
// loop through $data and create curl handles
// then add them to the multi-handle
foreach ($data as $id => $d) {
$curly[$id] = curl_init();
$url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
curl_setopt($curly[$id], CURLOPT_URL, $url);
curl_setopt($curly[$id], CURLOPT_HEADER, 0);
curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curly[$id], CURLOPT_USERAGENT, 'Chrome: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2');
// extra options?
if (!empty($options)) {
curl_setopt_array($curly[$id], $options);
}
curl_multi_add_handle($mh, $curly[$id]);
}
// execute the handles
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);
// get content and remove handles
foreach($curly as $id => $c) {
$result[$id][] = curl_getinfo($c, CURLINFO_EFFECTIVE_URL);
$result[$id][] = curl_getinfo($c, CURLINFO_HTTP_CODE);
$result[$id][] = curl_getinfo($c, CURLINFO_CONTENT_TYPE);
$url = curl_getinfo($c, CURLINFO_EFFECTIVE_URL);
// loop data again
foreach ($data as $id => $d){
if($url==$d['url']){ // only check current url data
$text = curl_exec($c);
$result[$id][] = strpos($text, $d['post']);
}
}
curl_multi_remove_handle($mh, $c);
}
// all done
curl_multi_close($mh);
return $result;
}
Can anyone advise on whether my solution is appropriate? Is there a more efficient/better way to perform my strpos() check?
Thanks

Use an array so you can relate all the urls, strings, and curl handles together:
$stuff = array(
0 => array('url' => 'google', 'text' => 'googletext', 'curl' => null)
1 => array('url' => 'yahoo', 'text' => 'yahootext', 'curl' => null)
etc..
);
foreach($stuff as $key => $info) {
$stuff[$key]['curl'] = curl_init($stuff[$key]['url']);
curl_multi_add_handle($mh, $stuff[$key]['curl']);
}
Then do a similar loop when you're processing the results.

Related

How to turn Curl response into Json so I can use it on the front-end?

I have a function which uses curl_multi to make several GET requests. However, I can't figure out how to turn the response to JSON so I could access it on the front end. Usually I use json_decode(), however, it is not working this time. Maybe it's because I'm trying to decode a whole array of strings instead of a singular string?. Currently the response looks like this on the front-end:
And here's my function:
public function getVisited($username){
$user = User::where('username', $username)->first();
$visitedPlaces = $user->visitedPlaces;
$finalPlaces = [];
foreach ($visitedPlaces as $visitedPlace) {
$url = "https://maps.googleapis.com/maps/api/place/details/json?placeid=" . $visitedPlace->place_id . "&key=AIzaSyDQ64lYvtaYYYNWxLzkppdN-n0LulMOf4Y";
array_push($finalPlaces, $url);
}
$result = array();
$curly = array();
$mh = curl_multi_init();
foreach ($finalPlaces as $id => $d) {
$curly[$id] = curl_init();
$url = $d;
curl_setopt($curly[$id], CURLOPT_URL, $url);
curl_setopt($curly[$id], CURLOPT_HEADER, 0);
curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curly[$id], CURLOPT_SSL_VERIFYPEER, 0); // Skip SSL Verification
curl_multi_add_handle($mh, $curly[$id]);
}
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);
foreach($curly as $id => $c) {
$result[$id] = curl_multi_getcontent($c);
curl_multi_remove_handle($mh, $c);
}
curl_multi_close($mh);
return response()->json([
'sights' => $result
], 201);
}
If you are expecting a json string from the curl_multi_getcontent($c) then you should decode it:
$result[$id] = json_decode(curl_multi_getcontent($c));
Then it should be encoded in the response appropriately.

Could not get any response while using curl in PHP

I am fetching the data using the curl in php but as many data are present it returning the 0 output. I am providing my code below.
$result = array();
// multi handle
$mh = curl_multi_init();
$idArr=[2,147,92];
foreach ($idArr as $key => $value) {
$fetchURL = "http://example.com/index.php/rest/V1/categories/".$value."/products/";
//echo $fetchURL.'<br>';
$multiCurl[$key] = curl_init();
curl_setopt($multiCurl[$key], CURLOPT_URL,$fetch_url);
curl_setopt($multiCurl[$key], CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($multiCurl[$key], CURLOPT_HEADER,array("Content-Type: application/json", "Authorization: Bearer " . json_decode($token)));
curl_setopt($multiCurl[$key], CURLOPT_RETURNTRANSFER,true);
curl_multi_add_handle($mh, $multiCurl[$key]);
}
$index=null;
do {
curl_multi_exec($mh,$index);
} while($index > 0);
// get content and remove handles
foreach($multiCurl as $k => $ch) {
$result[$k] = curl_multi_getcontent($ch);
curl_multi_remove_handle($mh, $ch);
}
// close
curl_multi_close($mh);
print_r($result);
Here I have to pass the multiple request and get the result but in this case no result is coming. While I am using simple curl the result is coming. Here my requirement is to reduce the response time.
You are missing the implementation of "curl_multi_select". I have done it for you but not tested. Give it a go
$result = array();
// multi handle
$mh = curl_multi_init();
$multiCurl = array();
$idArr=[2,147,92];
foreach ($idArr as $key => $value) {
$fetchURL = "http://example.com/index.php/rest/V1/categories/".$value."/products/";
//echo $fetchURL.'<br>';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$fetch_url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_HEADER,array("Content-Type: application/json", "Authorization: Bearer " . json_decode($token)));
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
$multiCurl[$key] = $ch;
curl_multi_add_handle($mh, $ch);
}
$active =null;
do {
$mrc = curl_multi_exec($mh,$active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
foreach($multiCurl as $k => $ch) {
$result[$k] = curl_multi_getcontent($ch);
curl_multi_remove_handle($mh, $ch);
}
// close
curl_multi_close($mh);
print_r($result);
I think you can't get response due to set wrong variable in curl option set.Your third line in foreach loop should be like that
curl_setopt($multiCurl[$key], CURLOPT_URL,$fetchURL);
also you write wrong syntex for headers option replace CURLOPT_HEADER by CURLOPT_HTTPHEADER
here is sample code that works perfectly
$idArr=[20,18,21];
$mh = curl_multi_init();
$requests = array();
$curl_objs_arr = [];
foreach ($idArr as $key => $cat) {
$fetchURL = "http:example.com/v2/products?category=".$cat;
$requests[$key] = curl_init($fetchURL);
curl_setopt($requests[$key], CURLOPT_URL,$fetchURL);
curl_setopt($requests[$key], CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($requests[$key], CURLOPT_HTTPHEADER,array("Content-Type: application/json","Authorization: Bearer " . json_decode($token)));
curl_setopt($requests[$key], CURLOPT_RETURNTRANSFER,true);
curl_multi_add_handle($mh, $requests[$key]);
}
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);
foreach ($requests as $key => $request) {
$result[$key] = curl_multi_getcontent($request);
curl_multi_remove_handle($mh, $request);
}
curl_multi_close($mh);
echo "<pre>";
print_r($result);exit;
set your code by this way, that's solve your problem.

How to capture curl_errno in curl_multi mode?

For some reason curl_errno($value) always returns 0 instead of 6 when I try url like stkovrflow.com. This is a non existing domain. So curl supposed to return 6. But i'm getting 0.
Can someone tell me what's wrong with my code?
This is how I check curl error
if (curl_errno($value) !== 0)
{
$handles[$key]['error_code'] = curl_errno($value);
}
Here is my full code
<?php
protected function curl($url)
{
$mh = curl_multi_init();
$handles = array();
foreach ($url as $link)
{
$handles[$link] = curl_init($link);
curl_setopt($handles[$link], CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($handles[$link], CURLOPT_HEADER, true);
curl_setopt($handles[$link], CURLOPT_RETURNTRANSFER, true);
curl_setopt($handles[$link],CURLOPT_FAILONERROR,true);
curl_setopt($handles[$link], CURLOPT_FOLLOWLOCATION, $this->curlFollowLocation);
curl_setopt($handles[$link], CURLOPT_MAXREDIRS, $this->curlMaxRedirects);
curl_setopt($handles[$link], CURLOPT_TIMEOUT, $this->curlTimeout);
curl_setopt($handles[$link], CURLOPT_USERAGENT, $this->curlUserAgent);
curl_setopt($handles[$link], CURLOPT_AUTOREFERER, true);
curl_multi_add_handle($mh, $handles[$link]);
}
$running = null;
do {
curl_multi_exec($mh, $running);
usleep(200000);
} while ($running > 0);
foreach ($handles as $key => $value)
{
$handles[$key] = false;
$handles[$key]['error_code'] = false;
if (curl_errno($value) !== 0)
{
$handles[$key]['error_code'] = curl_errno($value);
} else {
$response = curl_multi_getcontent($value);
$httpCode = curl_getinfo($value, CURLINFO_HTTP_CODE);
if ( $httpCode != 200 ) {
$handles[$key]['error_code'] = $httpCode;
} else {
$handles[$key]['html'] = $response;
}
}
curl_multi_remove_handle($mh, $value);
curl_close($value);
}
curl_multi_close($mh);
return $handles;
}
Update:
Looks like curl_errno does not work (see bug report) in curl multi mode. Instead we should use curl_multi_info_read. When I use curl_multi_info_read like this
$e_code = curl_multi_info_read($mh);
var_dump($e_code);
this is my var_dump output.
array (size=3)
'msg' => int 1
'result' => int 6
'handle' => resource(7, curl)
As you can it returns 6 correctly. But php doc says
The data the returned resource points to will not survive calling curl_multi_remove_handle().
Unfortunately my script depends on curl_multi_remove_handle(). Any solution?
=======

php retrieve curl_setopt POSTFIELDS string

Is there any way to retrieve the curl_setopt POSTFIELDS string which I had posted to the site after the curl_multi_exec($mh, $running) command?
Thanks.
You have to keep that data together with the individual resources:
$handles = array();
foreach ($urls as $url) {
$ch = curl_init($url);
$data = 'whatever';
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
$handles[$url] = array(
'ch' => $ch,
'data' => $data,
);
}
This keeps the cURL handles and data together in a single structure that you can later use to inspect.
foreach($handles as $url => $data) {
// $url is the page you requested for this particular handle
// $data['data'] contains the data that goes with it
$body = curl_multi_getcontent($data['ch']);
curl_multi_remove_handle($mh, $data['ch']);
curl_close($data['ch']);
}

PHP curl_multi_exec runs once

I'm having trouble creating multiple xml requests using php's curl_multi_exec.
The problem is that the do...while loop containing the curl_multi_exec command runs only once and then quits.
Resources Used:
http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/
http://php.net/manual/en/function.curl-multi-exec.php/
http://www.rustyrazorblade.com/2008/02/curl_multi_exec/
Take a look at my code:
//Multi handle curl initialization
$mh = curl_multi_init();
//set url
$url = 'my_url';
foreach($latLng as $id => $l) {
$ch[$id] = curl_init();
//$request previously set
//Initialize and set options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $request);
//add to multi_handle
curl_multi_add_handle($mh, $ch[$id]);
}
//Execute the handles
$running = null;
do {
$mrc = curl_multi_exec($mh, $running);
$ready=curl_multi_select($mh);
echo "Ran once\n";
} while ($mrc == CURLM_CALL_MULTI_PERFORM && $ready > 0);
while ($active && $mrc == CURLM_OK) {
if ($curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $running);
echo "Ran again\n";
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
foreach ($mh as $c) {
// HTTP response code
$code = curl_getinfo($c, CURLINFO_HTTP_CODE);
// cURL error number
$curl_errno = curl_errno($c);
// cURL error message
$curl_error = curl_error($c);
// output if there was an error
if ($curl_error) {
echo("*** cURL error: ($curl_errno) $curl_error\n");
}
}
//get content and remove handles
foreach ($ch as $c) {
$result[] = curl_multi_getcontent($c);
curl_multi_remove_handle($mh, $c);
}
print_r($result);
//Close curl
curl_multi_close($mh);
}
I know the request is valid because I receive the correct return data when I perform a single curl execution. The problem lies with the curl_multi_exec().
The output I am receiving is "Ran once" followed by the empty arrays of the curl_multi_getcontent() calls. See below:
Ran once
Array
(
[0] =>
[1] =>
[2] =>
[3] =>
[4] =>
[5] =>
[6] =>
[7] =>
[8] =>
)
Any help is greatly appreciated.
You're not setting up the curl options correctly:
Currently, you're setting options on $ch which is your array, you need to be setting the options specifically on the current curl handler, which in your loop is $ch[$id]:
//Initialize and set options
curl_setopt($ch[$id], CURLOPT_URL, $url);
curl_setopt($ch[$id], CURLOPT_HEADER, 0);
curl_setopt($ch[$id], CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch[$id], CURLOPT_POST, 1);
curl_setopt($ch[$id], CURLOPT_POSTFIELDS, $request);
change this:
foreach ($mh as $c) {
$code = curl_getinfo($c, CURLINFO_HTTP_CODE);
to:
for($i=1;$i<=count($array);$i++){
$code = curl_multi_getcontent($ch[$i]);
assuming $array is the array for your multiple $url.

Categories