PHP Curl Do While Loop on Pagination with Limit and Offset - php

I am newbie here and want to ask about PHP loop
I want to get 'itemid' from myendpoint and store it to $result_array
Myendpoint has pagination
First page, offset=0, second page, offset=100, etc
Maximum itemid on first page is 100, second page is 100, etc
The last page is unknown, so I set count($result_array) % 100 == 0 as while condition
The logic is if first page contain 100 itemid, do second page, if second page contain 100 itemid, do third page, etc
My script only work if first page contain < 100 itemid and there is no second page
How to solve this?
Thank you.
this is my script:
$limit = 100;
$offset = 0;
$result_array = array();
$url = 'https://myendpoint/?limit=' . $limit . '&offset=' . $offset;
do {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
$response = curl_exec($ch);
$data = json_decode($response, true);
curl_close($ch);
$itemIds = array();
foreach ($data["items"] as $row) {
$itemIds[] = $row["itemid"];
}
$result_array = array_merge($result_array, $itemIds);
$offset = $offset + $limit;
} while (count($result_array) % 100 == 0);
print_r($result_array);

Pretty simple error: you use the same URL over and over, as you build it outside your loop. Move it into your loop, such that it contains the $offset you are calculating within that loop

Related

Foreach loop inside while loop - it never ends

So, I have one curl API call which works fine when I do foreach outside the while loop. Once I move the foreach inside (because I need the values inside) it becomes an infinity loop.
This is the setup
$query = "SELECT id, vote FROM `administrators` WHERE type = 'approved'";
$result = $DB->query($query);
$offset = 0;
$length = 5000;
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
do {
curl_setopt($ch, CURLOPT_URL, "https://api.gov/data?api_key=xxxxxxxxxx&start=1960&sort[0][direction]=desc&offset=$offset&length=$length");
$jsonData = curl_exec($ch);
$response = json_decode($jsonData);
foreach($response->response->data as $finalData){
$allData[] = $finalData;
}
$offset += count($response->response->data);
} while ( count($response->response->data) > 0 );
curl_close($ch);
while($row = $DB->fetch_object($result)) {
foreach ( $allData as $key => $finalData1 ) {
// rest of the code
}
}
Once I run the page it goes infinity or until my browser crash. If I move foreach ( $allData as $key => $finalData1 ) { } outside the while(){} there is no such problem.
Any ideas on what can be the problem here?
UPDATE: // rest of the code
$dataValue = str_replace(array("--","(s)","NA"),"NULL",$finalData1->value);
if($frequency == "dayly") {
if($dataValue) {
$query = "UPDATE table SET $data_field = $dataValue WHERE year = $finalData1->period AND id = $row->id LIMIT 1";
}
}
if(isset($query))
$DB->query($query);
unset($query);
One of the issues could be that where
// rest of the code
is, you have duplicate variable names, thus overriding current positions in arrays and loops.
However, you should change your approach to something like
$rows = Array();
while($row = $DB->fetch_object($result)) $rows[] = $row;
foreach ($rows as $row) {
foreach ($allData as $key => $finalData1) {
// rest of the code
}
}
That way you can read resultset from database faster and free it before you continue.

PHP cURL requests being slow

im trying to grab the contents from a URL(which is a json) that changes for each iteration of my loop. The problem with my method of doing things is that it is very slow and if I do about 120 iterations it takes over 40sec.
Here is my code:
$GetFriendListUrl = "http://api.steampowered.com/ISteamUser/GetFriendList/v0001/?key=mykey&steamid=".$other_steamid."&relationship=friend";
$GET_GetFriendListUrl= file_get_contents($GetFriendListUrl);
$raw_ids = json_decode($GET_GetFriendListUrl , TRUE);
$count = count($raw_ids['friendslist']['friends']);
$ci = curl_init();
curl_setopt($ci, CURLOPT_RETURNTRANSFER, true);
for ($x = 0; $x <= $count; $x++) {
$friendslist = $raw_ids['friendslist']['friends'][$x]['steamid'];
curl_setopt($ci, CURLOPT_URL, "https://api.steampowered.com/ISteamUser/GetPlayerSummaries/v0002/?key=mykey&steamids=".$friendslist);
$cont = curl_exec($ci);
$contFull = json_decode($cont, true);
$steamname = $contFull['response']['players'][0]['personaname'];
$steamprofileurl = $contFull['response']['players'][0]['profileurl'];
$friendimage = $contFull['response']['players'][0]['avatar'];
$friendimageData = base64_encode(file_get_contents($friendimage));
echo '<img class="other_friendsteamimage" src="data:image/jpeg;base64,'.$friendimageData.'">';
echo "<a class='other_friendlabel' href='$steamprofileurl'>$steamname</a>";
echo "<br>";
}
curl_close($ci);
I cannot be sure of the format of the data returned by the api and I have no means of testing the following but in line with the comment I made and based upon the documentation it would appear that sending few requests but with each request dealing with 100 steamIDs you should save considerable amount of time.
/* get the intial data */
$url = "http://api.steampowered.com/ISteamUser/GetFriendList/v0001/?key=mykey&steamid=".$other_steamid."&relationship=friend";
$data= file_get_contents( $url );
$json = json_decode( $data );
$ids=array();
/* just grab the IDs and add to array - correct format to access records??? */
foreach( $json->friendslist->friends as $obj ){
$ids[]=$obj->steamid;
}
/* split the IDs into chunks of 100 */
$chunks=array_chunk( $ids, 100 );
/* send a request per chunk of 100 */
foreach( $chunks as $chunk ){
$url=sprintf('https://api.steampowered.com/ISteamUser/GetPlayerSummaries/v0002/?key=mykey&steamids=%s',implode(',',$chunk));
$curl = curl_init( $url );
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$res=curl_exec( $curl );
if( $res ){
$data=json_decode($res,true);
/* do stuff .... */
}
curl_close($curl);
}
echo 'Finito';

how to save output result outside the loop

How to use a php variable outside the loop
how to save output result outside the loop
this variable $jsonrs
this is the result I want it outside the loop
"1""1.jpg""2""2.jpg""3""3.jpg""4""4.jpg"
$url = 'https://hentaifox.com/gallery/58769/';
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($curl);
preg_match_all('!<img class="lazy no_image" data-src="(.*?)"!', $result, $manga_name);
$items = array();
foreach ($manga_name[1] as $key => $manganm) {
$imag_manga = str_replace('t.jpg','.jpg',$manganm);
$imagerep = 'https:'.$imag_manga;
$filename = basename($imagerep);
$imag_num = str_replace('.jpg','',$filename);
$array_name = array($imag_num => $filename);
$json1 = json_encode($imag_num);
$json2 = json_encode($filename);
$jsonrs = $json1.$json2;
print_r($jsonrs);
}
Your end result in $jsonrs is a bit unusual, I (am assuming) that you want to JSON encode the list of images, if so then use $items to keep a list of each image and then json_encode() this list after the loop...
foreach ($manga_name[1] as $key => $manganm) {
$filename = basename($manganm);
$imag_num = str_replace('t.jpg','',$filename);
$items[$imag_num] = $filename;
}
echo json_encode($items);
will give you
{"1":"1t.jpg","2":"2t.jpg","3":"3t.jpg","4":"4t.jpg"}

I need to load the distance data from Google Maps and save it to my Database

I got 5550 records of different routes and I need to do a foreach loop for each record and get the API data.
So I made a function with Guzzle in Laravel:
public function getDirectionDistance($origins, $distinations)
{
$client = new Client();
$res = $client->get("https://maps.googleapis.com/maps/api/distancematrix/json?origins=$origins&destinations=$distinations&key=ччч")->getBody()->getContents();
$obj = json_decode($res, true);
$distance = $obj['rows'][0]['elements'][0]['distance']['text'];
$clean = $string = str_replace(' km', '', $distance);
return $clean;
}
I used it in a store method
public function store()
{
$route = $this->route->with('from','to')->get();
$maps = new Maps();
foreach ($route as $item){
$direction = new Direction();
$from = $item->from->name;
$to = $item->to->name;
$direction->route_id = $item->id;
$direction->distance = $maps->getMapsApi("$from,israel","$to,israel");
$direction->save();
sleep(3);
}
}
But when I do It, I get 1 distance for 200 routes and then after 200 row I get the next distance for the next route. How to stop and wait for api to be completed, save it and then start the next row. I need the data to create a Machine Learning price calculator.
For me work in this way:
I created a function that make a call to google with CURL:
public function calculateDistance($origins, $destination){
$staticDistanceModel = New StaticDistance();
$url = "https://maps.googleapis.com/maps/api/distancematrix/json?origins=" . $origins . "&destinations=" . $destination . "&mode=driving&language=it-IT&key=xyz";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXYPORT, 3128);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
curl_close($ch);
$response_a = json_decode($response, true);
if (isset($response_a['rows'][0]['elements'][0]['distance']['value'])) {
$m = $response_a['rows'][0]['elements'][0]['distance']['value'];
}else{
$m = 0;
}
$staticDistanceModel->insertStatic($origins, $destination, $m);
}
}
the function in my model is something like:
public function insertStatic($origins, $destination, $m){
$arrayInsert = array('origins'=>$origins, 'destination'=>$destination,'distance'=>$m);
Self::create($arrayInsert);
}
And in my controller I have forEach() like this:
foreach ($array as $object) {
$calculator = $this->calculateDistance($object->origins, $object->destination);
}
But be careful because google limit request, and the time for 5500 records maybe long, so you can chunk array.
Hope this can help you

Simultaneous HTTP requests in PHP with cURL

I'm trying to take a rather large list of domains query the rank of each using the compete.com API as seen here -> https://www.compete.com/developer/documentation
The script I wrote takes a database of domains I populated and initiates a cURL request to compete for the rank of the website. I quickly realized that this was very slow because each request was being sent one at a time. I did some searching and came across this post-> http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/ which explains how to perform simultaneous HTTP requests in PHP with cURL.
Unfortunately that script will take an array of 25,000 domains and try to process them all at once. I found that batches of 1,000 work quite well.
Any idea how to send 1,000 queries to compete.com then wait for completion and send the next 1,000 until the array is empty? Here's what I'm workin with thus far:
<?php
//includes
include('includes/mysql.php');
include('includes/config.php');
//get domains
$result = mysql_query("SELECT * FROM $tableName");
while($row = mysql_fetch_array($result)) {
$competeRequests[] = "http://apps.compete.com/sites/" . $row['Domain'] . "/trended/rank/?apikey=xxx&start_date=201207&end_date=201208&jsonp=";
}
//first batch
$curlRequest = multiRequest($competeRequests);
$j = 0;
foreach ($curlRequest as $json){
$j++;
$json_output = json_decode($json, TRUE);
$rank = $json_output[data][trends][rank][0][value];
if($rank) {
//Create mysql query
$query = "Update $tableName SET Rank = '$rank' WHERE ID = '$j'";
//Execute the query
mysql_query($query);
echo $query . "<br/>";
}
}
function multiRequest($data) {
// array of curl handles
$curly = array();
// data to be returned
$result = array();
// multi handle
$mh = curl_multi_init();
// loop through $data and create curl handles
// then add them to the multi-handle
foreach ($data as $id => $d) {
$curly[$id] = curl_init();
$url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
curl_setopt($curly[$id], CURLOPT_URL, $url);
curl_setopt($curly[$id], CURLOPT_HEADER, 0);
curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);
// post?
if (is_array($d)) {
if (!empty($d['post'])) {
curl_setopt($curly[$id], CURLOPT_POST, 1);
curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']);
}
}
curl_multi_add_handle($mh, $curly[$id]);
}
// execute the handles
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);
// get content and remove handles
foreach($curly as $id => $c) {
$result[$id] = curl_multi_getcontent($c);
curl_multi_remove_handle($mh, $c);
}
// all done
curl_multi_close($mh);
return $result;
}
?>
Instead of
//first batch
$curlRequest = multiRequest($competeRequests);
$j = 0;
foreach ($curlRequest as $json){
You can do:
$curlRequest = array();
foreach (array_chunk($competeRequests, 1000) as $requests) {
$results = multiRequest($requests);
$curlRequest = array_merge($curlRequest, $results);
}
$j = 0;
foreach ($curlRequest as $json){
$j++;
// ...
This will split the large array into chunks of 1,000 and pass those 1,000 values to your multiRequest function which uses cURL to execute those requets.
https://github.com/webdevelopers-eu/ShadowHostCloak
This does exactly what you want. Just pass empty argument to new Proxy() to bypass proxy and make direct requests.
You can stuff 1000 requests in it and call $proxy->execWait() and it will process all requests simultaneously and exit that method when everything is done... Then you can repeat.

Categories