scraping "next pages" using PHP, cURL, simplehtmldom - php

Question

I finally got my scraper working (somewhat), but now I'd like to know how I can automatically go to the next page and scrape the same info from there. I'm using cURL to copy the entire page (otherwise I get a 500 error). Here's my code:
<?
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "http://example.com/results.asp?&j=t&page_no=1");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$html = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
// print $html . "\n";
require 'simple_html_dom.php';
$dom = new simple_html_dom();
$dom->load($html);
foreach($dom->find("div[#id='schoolsearch'] tr") as $data){
$tds = $data->find("td");
if(count($tds)==3){
$record = array(
'school' => $tds[1]->plaintext,
'city' => $tds[2]->plaintext
);
print json_encode($record) . "\n";
file_put_contents('schools.csv', json_encode($record) . "\n", FILE_APPEND);
}
}
?>
It's not perfect, but it's what works right now! Anyone know how I can move to the next pages?

Answer 1

Wrap it in a loop:
$maxPages = 10;
for ($i = 1; $i <= $maxPages; $i++) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://example.com/results.asp?&j=t&page_no=$i");
etc...
}
You'll need to tidy up a bit more,to avoiding including that file on every page, but you get the idea.