Paging through api - PHP - php

Question

I am trying to construct a php script that will page through an API. The api return ~25197 XML records. I am able to pass a start_offset and a end_offset to the API which will return a subset of the results.
The challenge I am having is that the for loop is not capturing the remaining records that are not within the 1000.
Example, the current for loop processes the records in blocks of 1000 (0-1000,1001-2000,2001-3000, etc.) I am not able to get the final block - 25,000 to 26,000. The for loop stop processing at 24,000 - 25,000. This leaves me with 197 unprocessed XML results.
<?php
//Set Start and Offset Parameters
$start_offset = 0;
$end_offset = 0;
$items_per_page = 1000;
$number = 0;
$counter = -2;
for ($count=0; $count<=100; $count++) {
$counter++;
//Validate that the counter is not null
if ($number != null){
echo "\n";
echo file_get_contents($static_url . "/sc_vuln_query-compliance.php?start=$start_offset&end=$end_offset&seq=$counter");
}
//Initialize the start and end offset variables
$end_offset = $number+=$items_per_page;
$start_offset = $number-$items_per_page+1;
//We want to start at record 0, reset start_offset back to 0 instead of 1
if($start_offset == 1) {
$start_offset = $number-$items_per_page;
}
// We are at the end of the total records, display the remaining
if ($number>$total_xml_records) {
$counter = $counter+1;
$padding = $end_offset + $items_per_page;
echo "\n";
echo file_get_contents($static_url. "/sc_vuln_query-compliance.php?start=$start_offset&end=$padding&seq=$counter");
break;
}
}
?>

Answer 1

Your code is at least buggy at this line $padding = $end_offset + $items_per_page; - here you raise the end for your last loop by another 1000 and therefor get ?start=25001&end=27000&seq=25.
Try $padding = $total_xml_records; instead, this will get you ?start=25001&end=25197&seq=25.
Anyway your code is quite complicated. Try this:
$total_xml_records = 25197; // index 0 .. 25196
$offset = 0;
$counter = 0;
while ($offset < $total_xml_records) {
echo "\n";
echo $static_url . "/sc_vuln_query-compliance.php?start=".($offset)."&end=".(min($offset+$items_per_page-1, $total_xml_records-1))."&seq=".($counter++);
$offset += $items_per_page;
}