Loop through paginated data JSON using PHP - php

Question

I'm working with a REST API that returns data in paginated format. 1 page will have 100 records. If there is more data there will be a "hasMore" parameter and "offset" parameter defined which you can use to retrieve the next page of results.
So far I've got the following code:
function getData(){
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "ENDPOINT",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$data = json_decode($response);
//if there is more get them.....
if($data->hasMore == 1){ //make another request...
while($data->hasMore == 1){
$offset = $data->offset;
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "ENDPOINT",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$data = json_decode($response);
$allData = array_merge($data->data, $data->data);
}
}
return $data; //return the results for use
}
My issue is that I'm struggling to have it check if there is more data and if there is request the next page all within the same function. I hope this makes sense and I'd appreciate any help or advice you can offer.
Right now I have it working as I know there is another page. The issue is how can I create it so that it will check the parameter to see if there is more and if there is another page request and append to the existing array of data.

Answer 1

Based on #TommyLee answer, but using the returned offset (just guessing how it is supposed to be used):
function getData($offset = 0){
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "ENDPOINT" . "?offset=" . $offset,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$data = json_decode($response);
return $data; //return the results for use
}
$allData = array();
do {
$offset = $data->offset ?: 0;
$data = getData($offset);
foreach($data->data as $row){
$allData[] = $row;
}
} while($data->hasMore == 1);
This way, if nothing is passed to getData(), it assumes the offset is 0, otherwise, you pass it the next offset from the last response.
Again, how the offset is used isn't clear, so you might need to adjust my example to fit the proper usage for $offset.

Answer 2

Your function is fine, but you should really separate the getData function and looping part.
function getData(){
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "ENDPOINT",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$data = json_decode($response);
return $data; //return the results for use
}
do{
$data = getData();
$i = 0;
for(i=0; i<data.length; i++){
//do data processing here
}
}while(data->hasMore == 1);