How to deal with large amount of data in API - php

I am developing a dashboard API for doing some operations related to users (create user, update, transfer money,...) by using PHP CURL to retrieve the data from an external source and display it to the front-end.
Everything was working well until the data started to grow at the database.
My senario is that each Admin after logging in to the dashboard should only do operations on his related users, that's why it takes too much time to retrieve all users from the source and then check the condition for related users and display them to this admin.
Is there a better solution for such kind of operations ?
I tried the following code to retrieve the data from the source and looped over requests since it's limited to 100 at each request.
function get_users($session_id){
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "THE_URL",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS =>"{\"offset\": 0,\"limit\": 100}",
CURLOPT_HTTPHEADER => array(
"x-api-token: TOKEN",
"Content-Type: application/json",
),
));
$response = curl_exec($curl);
$data = json_decode($response, true);
curl_close($curl);
$allcollection = [];
foreach ($data['users'] as $key => $value) {
if(isset($value['admin']['id'])){
if($value['admin']['id'] == $session_id){
array_push($allcollection,$value);
}
}}
$totalrows = $data['nrOfResults'];
$offset = 100;
while($totalrows >= 100){
$newcurl = curl_init();
curl_setopt_array($newcurl, array(
CURLOPT_URL => "THE_URL",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS =>"{\"offset\": ".$offset.",\"limit\": 100}",
CURLOPT_HTTPHEADER => array(
"x-api-token: token",
"Content-Type: application/json",
),
));
$newresponse = curl_exec($newcurl);
$newdata = json_decode($newresponse, true);
curl_close($newcurl);
foreach ($newdata['users'] as $key => $value) {
if(isset($value['admin']['id'])){
if($value['admin']['id'] == $session_id){
array_push($allcollection,$value);
}
}}
$totalrows = $totalrows - 100;
$offset = $offset + 100;
}
return $allcollection; }

Related

How to send French Characters in PHP CURL POST request?

I have French message in $sms variable and that I need to send it but I am getting error & while I using some English text, it will work fine. How to send French message in Curl PHP.
I have used "Content-Type: application/x-www-form-urlencoded; charset=utf-8"
below is my code:
$sms='{"text":{"fr":"Bienvenue sur Gamebar! Accédez au service et gérer votre abonnement sur"}} ';
$ordered_params = array(
'order' => '112423',
'prompt_content_args'=>$sms
);
$to_sign = '';
foreach ($ordered_params as $v)
{
$to_sign .= $v;
}
// maybe use utf8_encode($to_sign) if your php file is encoded iso-8859-1
$digest = hash_hmac('sha256', $to_sign, '8awIiFHqV8zlimG');
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://url.com", // no real
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => "digest=$digest&order=112423&prompt_content_args=$sms",
CURLOPT_HTTPHEADER => array(
"Content-Type: application/x-www-form-urlencoded; charset=utf-8"
),
));
$result = curl_exec($curl);
There are more problems in that code than could be addressed in comments:
$sms_data = [
"text" => [
// make sure that this is UTF-8 encoded _beforehand_. Check your editor's settings and/or source data.
"fr" => "Bienvenue sur Gamebar! Accédez au service et gérer votre abonnement sur"
]
];
$sms = json_encode($sms_data);
$ordered_params = [
'order' => '112423',
'prompt_content_args' => $sms
];
$to_sign = '';
foreach ($ordered_params as $v) {
$to_sign .= $v;
}
$digest = hash_hmac('sha256', $to_sign, '8awIiFHqV8zlimG');
// consistent data structure
$curl_data = [
'digest' => $digest,
'order' => 112423,
'prompt_content_args' => $sms
];
$curl = curl_init();
curl_setopt_array($curl, [
CURLOPT_URL => "https://url.com", // no real
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_POST => true, // why on earth were you using CUSTOMREQUEST?
CURLOPT_POSTFIELDS => $curl_data, // let curl handle the encoding
CURLOPT_HTTPHEADER => [
"Content-Type: application/x-www-form-urlencoded; charset=utf-8"
],
]
);
If, for some reason, you need to url-encode a set of parameters outside of a cURL call, you would want to use http_build_query().

Grab youtube playlists data including each video within playlist using nextpage

Im trying to work out on how to grab youtube data from the API using PHP/CRUL to list out all the youtube playlists within the my channel id and then grab each video data within each playlist.
Playlist 1: Cars
Video 1: BMW - duration 10 mins
Video 2: Ford - duration 05 mins
Playlist 2: Bikes
Video 1: Honda - duration 03 mins
Video 2: Yamaha - duration 03 mins
Video 3: Kawasaki - duration 02 mins
etc
There are over 50 playlists and some with over 50 videos within a playlist. I just cant work out to use the nextPageToken. How should i go about this, are there any classes available that i can use?
Thank you for any advice.
function grabPLAYLISTS($pageToken){
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://www.googleapis.com/youtube/v3/playlists?part=snippet,contentDetails&channelId=&maxResults=10&key=&pageToken=".$pageToken,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
return json_decode($response, true);
}
function grabITEMS($therequestedlist, $pageToken){
$curl2 = curl_init();
curl_setopt_array($curl2, array(
CURLOPT_URL => "https://www.googleapis.com/youtube/v3/playlistItems?part=snippet,contentDetails&key=&playlistId=".$therequestedlist."&maxResults=10&pageToken=".$pageToken,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache"
),
));
$response2 = curl_exec($curl2);
$err2 = curl_error($curl2);
curl_close($curl2);
return json_decode($response2, true);
}
function grabVIDEO($videoID){
$curl3 = curl_init();
curl_setopt_array($curl3, array(
CURLOPT_URL => "https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=".$videoID."&key=",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache"
),
));
$response3 = curl_exec($curl3);
$err3 = curl_error($curl3);
curl_close($curl3);
return json_decode($response3, true);
}
$playlistdata = grabPLAYLISTS();
$counter_playlistdata = count($playlistdata['items']) -1;
$totalResults = $playlistdata['pageInfo']['totalResults'];
$resultsPerPage = $playlistdata['pageInfo']['resultsPerPage'];
for ($x = 0; $x <= $counter_playlistdata; $x++) {
$therequestedlist = $playlistdata['items'][$x]['id'];
echo "list ".$x."-".$therequestedlist."<br>";
}

Loop through paginated data JSON using PHP

I'm working with a REST API that returns data in paginated format. 1 page will have 100 records. If there is more data there will be a "hasMore" parameter and "offset" parameter defined which you can use to retrieve the next page of results.
So far I've got the following code:
function getData(){
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "ENDPOINT",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$data = json_decode($response);
//if there is more get them.....
if($data->hasMore == 1){ //make another request...
while($data->hasMore == 1){
$offset = $data->offset;
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "ENDPOINT",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$data = json_decode($response);
$allData = array_merge($data->data, $data->data);
}
}
return $data; //return the results for use
}
My issue is that I'm struggling to have it check if there is more data and if there is request the next page all within the same function. I hope this makes sense and I'd appreciate any help or advice you can offer.
Right now I have it working as I know there is another page. The issue is how can I create it so that it will check the parameter to see if there is more and if there is another page request and append to the existing array of data.
Based on #TommyLee answer, but using the returned offset (just guessing how it is supposed to be used):
function getData($offset = 0){
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "ENDPOINT" . "?offset=" . $offset,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$data = json_decode($response);
return $data; //return the results for use
}
$allData = array();
do {
$offset = $data->offset ?: 0;
$data = getData($offset);
foreach($data->data as $row){
$allData[] = $row;
}
} while($data->hasMore == 1);
This way, if nothing is passed to getData(), it assumes the offset is 0, otherwise, you pass it the next offset from the last response.
Again, how the offset is used isn't clear, so you might need to adjust my example to fit the proper usage for $offset.
Your function is fine, but you should really separate the getData function and looping part.
function getData(){
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "ENDPOINT",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$data = json_decode($response);
return $data; //return the results for use
}
do{
$data = getData();
$i = 0;
for(i=0; i<data.length; i++){
//do data processing here
}
}while(data->hasMore == 1);

WooCommerce REST API - Get all products with its variations details in a single api request

I am new to woo REST API,
i have a php page where i need to populate all products and variations. so i am using curl https://my_ip/index.php/wp-json/wc/v2/products then im looping it and passing product id to get variations so inside loop im using https://my_ip/index.php/wp-json/wc/v2/products/$variable_product_id/variations/ but it takes more time load because each product have more variations so is there any way get all products and variations in a single request?
like: is there any customisation can be made in class-wc-rest-products-controller.php or in any query or any where in woocommerce? so that i can get the result.
mycode:
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://my_ip/index.php/wp-json/wc/v2/products",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_SSL_VERIFYPEER=> false,
CURLOPT_HTTPHEADER => array(
"Authorization: Basic *****************",
"Cache-Control: no-cache"
),
));
$response = curl_exec( $curl) ;
curl_close( $curl );
$Product = json_decode( $response );
//after getting all product. looping all to get variations price.
foreach ( $Product as $row ){
$product_id= $row->id;
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://my_ip/index.php/wp-json/wc/v2/products/".$product_id."/variations/",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_SSL_VERIFYPEER=> false,
CURLOPT_HTTPHEADER => array(
"Authorization: Basic **************",
"Cache-Control: no-cache"
),
));
$response1 = curl_exec( $curl );
curl_close( $curl );
}
$variations=json_decode( $response1 );
print_r( $variations->price );
// here getting variation of particular product
so when we call https://my-ip/index.php/wp-json/wc/v2/products/423 we get all product and only variations id belongs to it as
"variations": [
603,
604,
605,
606
],
but what i need is details(price) of each variations
{
"id": 603,
"date_created": "2018-06-18T12:45:58",
"date_created_gmt": "2018-06-18T12:45:58",
"date_modified": "2018-06-29T06:41:21",
"date_modified_gmt": "2018-06-29T06:41:21",
"description": "",
"sku": "",
"price": "550",
"regular_price": "550",
}
note: i am creating a separate test.php page in /var/www/html/ . this page is not inside any wp-content
I did this using their phpLib
while (count($woocommerce->get('products',array('per_page' => 100, 'page' => $page))) > 0) {
$all_products = array_merge($all_products,$woocommerce->get('products',array('per_page' => 100, 'page' => $page)));
$page++;
}
if ($source_product->type = "variable") {
$variation = $woocommerce->get('products/'.$source_product->id.'/variations');
foreach ($variation as $source_child) {
//do stuff
}
}
every time in a loop you rewrite the value - response1
if you want save all value you need add this value in array
for example:
<?php
$array = array();
foreach ($Product as $row){
$product_id= $row->id;
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://my_ip/index.php/wp-json/wc/v2/products/" . $product_id . "/variations/",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_SSL_VERIFYPEER=> false,
CURLOPT_HTTPHEADER => array(
"Authorization: Basic **************",
"Cache-Control: no-cache"
),
));
$array[] = json_decode( curl_exec( $curl ) );
curl_close( $curl );
}
print_r( $array );

My foreach won't make a call for every coordinate that I pass

I pass some coordinates to my foreach to a reverse geocoding. I get only one coordinate but not them all. I think it's a problem of delay.
Here's my code:
foreach ($coordPointsArray as $coordPointArray) {
$latLangCoords = implode(",", $coordPointArray);
echo($latLangCoords);
$callToGoogleGeocode = curl_init();
$urlGeocode = 'https://maps.googleapis.com/maps/api/geocode/json?latlng='. urlencode($latLangCoords). '&key=' . $googleApiKey;
curl_setopt_array($callToGoogleGeocode, [
CURLOPT_URL => $urlGeocode,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => [
"cache-control: no-cache"
],
]);
$rawResponseGeocode = curl_exec($callToGoogleGeocode);
}
return($rawResponseGeocode);
here's the echo response:
45.4807897,9.21359845.4801858,9.214740945.4801858,9.214740945.4763715,9.21103245.4763715,9.21103245.4748449,9.211080445.4748449,9.211080445.4685044,9.211260645.4685044,9.211260645.4673681,9.211252645.4673681,9.211252645.4621453,9.211394845.4621453,9.211394845.4621861,9.20949345.4621861,9.20949345.4604364,9.2092372
I did a mistake. I was overwriting the variable at the end. I needed to store my responses in an array with array push.

Categories