PHP web scraping HTMLDOM pagination - php

Question

I am scraping this url as it is my final year project but this code only scrape 1 page of searched query I want pagination (like 1,2,3,4,5) at the end please help
I have implemented one data scraping script which fetch data using CURL.
But that fetch record only one page but i want all data because on that page pagination is there.
<form action="" method="post" class="form-horizontal" id="home-search">
<input type="text" name="keyword" id="keyword">
<input type="submit">
</form>
<?php
if(isset($_POST['keyword'])){
$keyword = urlencode($_POST['keyword']);
ini_set('display_errors', 1);
ini_set('max_execution_time', 300);
$html = file_get_contents('https://www.bestjobs.co.za/jobs/?q='.$keyword);
//echo $html;
$indeedDotPk = array();
//$html = file_get_contents($result);
libxml_use_internal_errors( true);
$doc = new DOMDocument;
$doc->loadHTML($html);
$xpath = new DOMXpath( $doc);
$node = $xpath->query( '//div[#class="paginas"]/ul/li/a/#href');
$total_pages = 0;
$start = 0;
$job_title_index = 0;
$job_link_index = 0;
$job_description_index = 0;
$job_experience_index = 0;
foreach ($node as $key => $value) {
$total_pages++;
// echo $value->textContent;
// echo "<br>";
// echo "<br>";
// echo "<br>";
}
for ($i=0; $i < $total_pages; $i++) {
ini_set('max_execution_time', 300);
$html = file_get_contents('https://www.bestjobs.co.za/jobs/?q='.$keyword.'&start='.$start);
libxml_use_internal_errors( true);
$doc = new DOMDocument;
$doc->loadHTML($html);
$xpath = new DOMXpath( $doc);
// Job Description
$node = $xpath->query('//a[#class="js-o-link"]');
foreach ($node as $key => $value) {
if(is_string($value->textContent)){
$indeedDotPk[$job_description_index++]['job_description'] = $value->textContent;
}
}
// Job Description
$start = $start + 10;
}
foreach ($indeedDotPk as $key => $value) {
if(!empty($value['job_description'])){
?>
<table border="1">
<tr >
<td>
</td>
<td>
</td>
<td>
</td>
<td>
<?php echo $value['job_description']?>
</td>
</tr>
Does anyone have an idea how I can set pagination in the end like 1,2,3,4,5 ?
If anyone has any suggestion then please help me.
Thanks...

Answer 1

Pass the paging parameter in the url like this
https://www.bestjobs.co.za/jobs/?q=sales&p=2
Wrap everything in a function and using for loop pass the paging parameter to the function like this
function webScrape($p){
//scraping code
}
for($i=0;$i>=100;$i++){
webScrape($i);
}