PHP Scraping - Loop through the pagination and extract data from each page - php

Question

I want to extract some data from local ecommerce site, emag.ro (more precisely, all products of a certain category - this involve that the script should run through the site pagination).
The facts:
each page contains maximum 60 products
first category page is https://www.emag.ro/telefoane-mobile/c
after the first page increments like this: https://www.emag.ro/telefoane-mobile/p2/c
(p2, p3 and so on)
I have untill now the following code:
<?php
$categoryPageUrl = 'https://www.emag.ro/telefoane-mobile/p{page_id}/c';
$products = [];
$productsPerPage = 60;
function calculateProductIndex($page_id, $product_index){
global $productsPerPage;
return ($productsPerPage * ($page_id - 1)) + $product_index;
}
// loop all category pages
for($i=1; $i<=1; $i++){
$categoryUrl = str_replace("{page_id}", $i, $categoryPageUrl);
$pageSrc = getRequest($categoryUrl);
$pageXPath = getXpathObject($pageSrc);
// get product title
$titleXpath = $pageXPath->query('//h2/a');
for($j = 0; $j < $titleXpath->length; $j++){
$position = calculateProductIndex($i, $j);
$title = $titleXpath->item($j)->nodeValue;
$products[$position]['name'] = $title;
}
}
// testing the output
print_r($products);
The issue where i am stuck is that i cannot get after the first page.
$products array is only returning 60 product titles (meaning it scrapes only the first page).
What i am doing wrong here and how can i loop through the pagination?

Answer 1

As mentioned in the comments, you need to first retrieve the total number of pages so your loop can iterate over that, instead of 1.
To do it, you can scrab the data from the first page:
$firstPageUrl = 'https://www.emag.ro/telefoane-mobile/p1/c';
$domDocument = new \DOMDocument();
$domDocument->loadHTMLFile($firstPageUrl, LIBXML_NOWARNING | LIBXML_NOERROR);
$xpath = new \DOMXPath($domDocument);
$lastPage = $xpath->evaluate('string((//a[#data-page])[position()=last()-1])');