PHP cURL requests being slow - php

im trying to grab the contents from a URL(which is a json) that changes for each iteration of my loop. The problem with my method of doing things is that it is very slow and if I do about 120 iterations it takes over 40sec.
Here is my code:
$GetFriendListUrl = "http://api.steampowered.com/ISteamUser/GetFriendList/v0001/?key=mykey&steamid=".$other_steamid."&relationship=friend";
$GET_GetFriendListUrl= file_get_contents($GetFriendListUrl);
$raw_ids = json_decode($GET_GetFriendListUrl , TRUE);
$count = count($raw_ids['friendslist']['friends']);
$ci = curl_init();
curl_setopt($ci, CURLOPT_RETURNTRANSFER, true);
for ($x = 0; $x <= $count; $x++) {
$friendslist = $raw_ids['friendslist']['friends'][$x]['steamid'];
curl_setopt($ci, CURLOPT_URL, "https://api.steampowered.com/ISteamUser/GetPlayerSummaries/v0002/?key=mykey&steamids=".$friendslist);
$cont = curl_exec($ci);
$contFull = json_decode($cont, true);
$steamname = $contFull['response']['players'][0]['personaname'];
$steamprofileurl = $contFull['response']['players'][0]['profileurl'];
$friendimage = $contFull['response']['players'][0]['avatar'];
$friendimageData = base64_encode(file_get_contents($friendimage));
echo '<img class="other_friendsteamimage" src="data:image/jpeg;base64,'.$friendimageData.'">';
echo "<a class='other_friendlabel' href='$steamprofileurl'>$steamname</a>";
echo "<br>";
}
curl_close($ci);

I cannot be sure of the format of the data returned by the api and I have no means of testing the following but in line with the comment I made and based upon the documentation it would appear that sending few requests but with each request dealing with 100 steamIDs you should save considerable amount of time.
/* get the intial data */
$url = "http://api.steampowered.com/ISteamUser/GetFriendList/v0001/?key=mykey&steamid=".$other_steamid."&relationship=friend";
$data= file_get_contents( $url );
$json = json_decode( $data );
$ids=array();
/* just grab the IDs and add to array - correct format to access records??? */
foreach( $json->friendslist->friends as $obj ){
$ids[]=$obj->steamid;
}
/* split the IDs into chunks of 100 */
$chunks=array_chunk( $ids, 100 );
/* send a request per chunk of 100 */
foreach( $chunks as $chunk ){
$url=sprintf('https://api.steampowered.com/ISteamUser/GetPlayerSummaries/v0002/?key=mykey&steamids=%s',implode(',',$chunk));
$curl = curl_init( $url );
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$res=curl_exec( $curl );
if( $res ){
$data=json_decode($res,true);
/* do stuff .... */
}
curl_close($curl);
}
echo 'Finito';

Related

PHP Curl Do While Loop on Pagination with Limit and Offset

I am newbie here and want to ask about PHP loop
I want to get 'itemid' from myendpoint and store it to $result_array
Myendpoint has pagination
First page, offset=0, second page, offset=100, etc
Maximum itemid on first page is 100, second page is 100, etc
The last page is unknown, so I set count($result_array) % 100 == 0 as while condition
The logic is if first page contain 100 itemid, do second page, if second page contain 100 itemid, do third page, etc
My script only work if first page contain < 100 itemid and there is no second page
How to solve this?
Thank you.
this is my script:
$limit = 100;
$offset = 0;
$result_array = array();
$url = 'https://myendpoint/?limit=' . $limit . '&offset=' . $offset;
do {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
$response = curl_exec($ch);
$data = json_decode($response, true);
curl_close($ch);
$itemIds = array();
foreach ($data["items"] as $row) {
$itemIds[] = $row["itemid"];
}
$result_array = array_merge($result_array, $itemIds);
$offset = $offset + $limit;
} while (count($result_array) % 100 == 0);
print_r($result_array);
Pretty simple error: you use the same URL over and over, as you build it outside your loop. Move it into your loop, such that it contains the $offset you are calculating within that loop

Access multiple URL at once in Curl PHP

I am working on API that return single currency record in one request. One request take 0.5-1 sec to response, and 15 requests take 7-15 seconds.
As i know server can manage 100s of request per seconds.
I want to hit 15 request on server at once so server will give response in 1-2 seconds not in 15 Seconds. Return all data in one single array to save my loading time.
Check my Code
I am using Loop, loop wait until previous curl request not complete. How can i say to loop, keep continue and dont wait for response.
$time_Start = microtime(true);
$ids = array(1,2,11,15,20,21); // 6 ids in demo, 15+ ids in real
$response = array();
foreach ($ids as $key => $id) {
$response[$id] = get_data($id);
}
echo "Time: ". (microtime(true)-$time_Start)."sec";
// output 5 seconds on 6 request
function get_data($id){
$fcs_api_key = "API_KEY";
$ch=curl_init();
curl_setopt($ch,CURLOPT_URL,"https://fcsapi.com/api/forex/indicators?id=".$id."&period=1d&access_key=".$fcs_api_key);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$buffer = curl_exec($ch);
curl_close($ch);
return $buffer;
}
You can use PHP multi curl https://www.php.net/manual/en/function.curl-multi-init.php
Below I write a code that open Parallel request.
$time_Start = microtime(true);
$ids = array(1,2,3,4,5,6); // You forex currency ids.
$response = php_curl_multi($ids);
echo "Time: ". (microtime(true)-$time_Start)."sec";
// Time: 0.7 sec
Function
function php_curl_multi($ids){
$parameters = "/api/forex/indicators?period=1d&access_key=API_KEY&id="; // ID will set dynamic
$url = "https://fcsapi.com".$parameters;
$ch_index = array(); // store all curl init
$response = array();
// create both cURL resources
foreach ($ids as $key => $id) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url.$id);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$ch_index[] = $ch;
}
//create the multiple cURL handle
$mh = curl_multi_init();
//add the handles
foreach ($ch_index as $key => $ch) {
curl_multi_add_handle($mh,$ch);
}
//execute the multi handle
do {
$status = curl_multi_exec($mh, $active);
if ($active) {
curl_multi_select($mh);
}
} while ($active && $status == CURLM_OK);
//close the handles
foreach ($ch_index as $key => $ch) {
curl_multi_remove_handle($mh, $ch);
}
curl_multi_close($mh);
// get all response
foreach ($ch_index as $key => $ch) {
$response[] = curl_multi_getcontent($ch);
}
return $response;
}

how to save output result outside the loop

How to use a php variable outside the loop
how to save output result outside the loop
this variable $jsonrs
this is the result I want it outside the loop
"1""1.jpg""2""2.jpg""3""3.jpg""4""4.jpg"
$url = 'https://hentaifox.com/gallery/58769/';
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($curl);
preg_match_all('!<img class="lazy no_image" data-src="(.*?)"!', $result, $manga_name);
$items = array();
foreach ($manga_name[1] as $key => $manganm) {
$imag_manga = str_replace('t.jpg','.jpg',$manganm);
$imagerep = 'https:'.$imag_manga;
$filename = basename($imagerep);
$imag_num = str_replace('.jpg','',$filename);
$array_name = array($imag_num => $filename);
$json1 = json_encode($imag_num);
$json2 = json_encode($filename);
$jsonrs = $json1.$json2;
print_r($jsonrs);
}
Your end result in $jsonrs is a bit unusual, I (am assuming) that you want to JSON encode the list of images, if so then use $items to keep a list of each image and then json_encode() this list after the loop...
foreach ($manga_name[1] as $key => $manganm) {
$filename = basename($manganm);
$imag_num = str_replace('t.jpg','',$filename);
$items[$imag_num] = $filename;
}
echo json_encode($items);
will give you
{"1":"1t.jpg","2":"2t.jpg","3":"3t.jpg","4":"4t.jpg"}

I need to load the distance data from Google Maps and save it to my Database

I got 5550 records of different routes and I need to do a foreach loop for each record and get the API data.
So I made a function with Guzzle in Laravel:
public function getDirectionDistance($origins, $distinations)
{
$client = new Client();
$res = $client->get("https://maps.googleapis.com/maps/api/distancematrix/json?origins=$origins&destinations=$distinations&key=ччч")->getBody()->getContents();
$obj = json_decode($res, true);
$distance = $obj['rows'][0]['elements'][0]['distance']['text'];
$clean = $string = str_replace(' km', '', $distance);
return $clean;
}
I used it in a store method
public function store()
{
$route = $this->route->with('from','to')->get();
$maps = new Maps();
foreach ($route as $item){
$direction = new Direction();
$from = $item->from->name;
$to = $item->to->name;
$direction->route_id = $item->id;
$direction->distance = $maps->getMapsApi("$from,israel","$to,israel");
$direction->save();
sleep(3);
}
}
But when I do It, I get 1 distance for 200 routes and then after 200 row I get the next distance for the next route. How to stop and wait for api to be completed, save it and then start the next row. I need the data to create a Machine Learning price calculator.
For me work in this way:
I created a function that make a call to google with CURL:
public function calculateDistance($origins, $destination){
$staticDistanceModel = New StaticDistance();
$url = "https://maps.googleapis.com/maps/api/distancematrix/json?origins=" . $origins . "&destinations=" . $destination . "&mode=driving&language=it-IT&key=xyz";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXYPORT, 3128);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
$response_a = json_decode($response, true);
if (isset($response_a['rows'][0]['elements'][0]['distance']['value'])) {
$m = $response_a['rows'][0]['elements'][0]['distance']['value'];
}else{
$m = 0;
}
$staticDistanceModel->insertStatic($origins, $destination, $m);
}
}
the function in my model is something like:
public function insertStatic($origins, $destination, $m){
$arrayInsert = array('origins'=>$origins, 'destination'=>$destination,'distance'=>$m);
Self::create($arrayInsert);
}
And in my controller I have forEach() like this:
foreach ($array as $object) {
$calculator = $this->calculateDistance($object->origins, $object->destination);
}
But be careful because google limit request, and the time for 5500 records maybe long, so you can chunk array.
Hope this can help you

Getting Php Curl Error

I want to grab first 10 results of that any url i passed it to function as parameter ..and i want to make data scraper for some sites
Getting Syntax error when I print the result on screen
syntax error on this line I don't why it giving me syntax error kindly help me....
print_r( $dse->crawl()->parse() );
<?php
class CURL_CRAWLER{
public $url;
public $request_type;
public $data;
public $post_params;
function __construct($url = '' , $request_type = 'GET')
{
$this->url = $url;
$this->request_type = $request_type;
$this->data = '';
$this->post_params = array();
}
/**crawl a document **/
function crawl()
{
$curl = curl_init( $this->url );
curl_setopt($curl, CURLOPT_HEADER, false);
curl_setopt($curl, CURLOPT_TIMEOUT, 60);
curl_setopt($curl, CURLOPT_USERAGENT, 'cURL PHP');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$this->data = curl_exec($curl);
curl_close($curl);
return $this; //make it a chainable method
}
/** Parse result data **/
function parse(){
$result = array();
$count = 0;
$dom = new DOMDocument;
$dom->preserveWhiteSpace = false;
$dom->loadHTML($this->data);
$xpath = new DOMXPath($dom);
$news = $xpath->query('//td[#bgcolor="#DDDDDD"]/table/tr[position()=2]/td[position()=2]');
foreach( $news as $n){
$result[] = $n->nodeValue;
$count++;
if ($count >9)
break; //we just need 10 results. Index starts from 0
}
return $result;
}
}
error_reporting(0);
$dse = new CURL_CRAWLER('http://www.dsebd.org/display_news.php');
echo "<pre>";
print_r( $dse->crawl()->parse() );
echo "<pre>";
?>
Your syntax error is that you should use explicit "greater than" sign instead of HTML entities > - server doesn't need those, it is not a browser that can render it correctly. Just change:
print_r( $dse->crawl()->parse() );
^^^^ ^^^^
to:
print_r( $dse->crawl()->parse() );

Categories