cURL PHP multiple requests for a dynamic URL (Looping Unknown Times) - php

Question

I'm new to cURL & I'm trying to retrieve data from a dynamic URL. I've successfully retrieved data from 1 page only, but what I want is retrieving data from all pages. The problem here is that pages are variable; I don't know how many time the code should loop to loop over all pages. i.e. the number of pages varies from one case to another, and a good program is the one that works in as many cases as possible. Consequently, putting the links in an array and looping over them isn't the right solution for this.
Here's a quick explanation of the URL I'm trying to retrieve data from:
https://link-search.api.cj.com/v2/link-search?website-id=[Your-ID]&link-type=banner
&advertiser-ids=1513033&records-per-page=100&page-number=' . $num
Did you notice the last variable $num? That should represent the number of the page from where data will be retrieved from. In some cases, it could be only 1, and in some other cases, it could be 10, 12 or 15 (it varies) depending on the parameters I choose, whether I want to see everything or filter some information.
Now here is the problem. How to dynamically increment that number as long as the request returns data? and if not, cURL should stop running?
Here's the code:
<?php
$num = 1;
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "XGET");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($curl, CURLOPT_ENCODING, "UTF-8");
curl_setopt($curl, CURLOPT_URL, 'https://link-search.api.cj.com/v2/link-search?website-id=[Your-ID]&link-type=banner&advertiser-ids=1513033&records-per-page=100&page-number=' . $num);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_COOKIE,true);
$result = curl_exec($curl);
$xml = simplexml_load_string($result) or die("Error: Cannot create object");
if (curl_errno($curl)) {
echo 'Error:' . curl_error($curl);
}
curl_close($curl);
?>

Answer 1

You can define your curl request inside the function and call that function like this:
<?php
$GLOBALS['num'] = 1;
function curlRequest()
{
// HERE DEFINE YOUR CURL REQUEST
// https://yourUrl.com?$GLOBALS['num']
if (curl_errno($curl)) {
echo 'Error:' . curl_error($curl);
// Exit from function in case there is no output
return;
} else {
$GLOBALS['num']++;
// Call the function to fetch NEXT page
curlRequest();
}
}
// Call the function for first time
curlRequest();
Assuming that curl will fire error if page number not exist or you can implement if condition from the result you are getting. Hope you will get some idea from this.
The main part here is if & else. You should implement if condition on the output you are getting and if page not exist then output will be different so you can simply return from the function in this case.