Finding nested links with Simple HTML DOM (recursion) - php

Question

I'm newbie about programming, so this is my issue. I am trying to build a recursive php spider usind Simple HTML DOM Parser, crawling into a certain website and returning a list of pages including 2xx, 3xx, 4xx and 5xx. I've been searching several days for a solution but (maybe due to my low experience) I haven't found anything working. My actual code finds all the links on the root/index page, however i would like to be able to find links inside those previously found links recursively and so on, for example to level 5. Assuming the root page is level 0, the recursive function I wrote only shows me level 1 links, repeating them 5 times. Any help appreciated. Thanks.
<?php
echo "<strong><h1>Sitemap</h1></strong><br>";
include_once('simple_html_dom.php');
$url = "http://www.gnet.it/";
$html = new simple_html_dom();
$html->load_file($url);
echo "<strong><h2>Int Links</h2></strong><br>";
foreach($html->find("a") as $a)
{
if((!(preg_match('#^(?:https?|ftp)://.+$#', $a->href)))&&($a->href != null)&&($a->href != "javascript:;")&&($a->href != "#"))
{
echo "<strong>" . $a->href . "</strong><br>";
}
}
echo "<strong><h2>Ext Links</h2></strong><br>";
foreach($html->find("a") as $a)
{
if(((preg_match('#^(?:https?|ftp)://.+$#', $a->href)))&&($a->href != null)&&($a->href != "javascript:;")&&($a->href != "#"))
{
echo "<strong>" . $a->href . "</strong><br>";
}
}
//recursion
$depth = 1;
$maxDepth = 5;
$recurl = "$a->href";
$rechtml = new simple_html_dom();
$rechtml->load_file($recurl);
while($depth <= $maxDepth){
echo "<strong><h2>Link annidati livello $depth</h2></strong><br>";
foreach($rechtml->find("a") as $a)
{
if(($a->href != null))
{
echo "<strong>" . $a->href . "</strong><br>";
}
}
$depth++;
}
//csv
echo "<strong><h1>Google Crawl Errors from CSV</h1></strong><br>";
echo "<table>\n\n";
$f = fopen("CrawlErrors.csv", "r");
while (($line = fgetcsv($f)) !== false) {
echo "<tr>";
foreach ($line as $cell) {
echo "<td>" . htmlspecialchars($cell) . "</td>";
}
echo "</tr>\n";
}
fclose($f);
echo "\n</table>";
?>

Answer 1

Try this:
I call this routine in a basic scraper to recursively find all of the links across the site. You'll have to put in some logic to prevent it from crawling external sites that are linked to from pages on your site, else you'll be running forever!
Note, I did get the majority of this code from another SO thread a while back, so the answers are out there.
function crawl_page($url, $depth = 2){
// strip trailing slash from URL
if(substr($url, -1) == '/') {
$url= substr($url, 0, -1);
}
// which URLs have we already crawled?
static $seen = array();
if (isset($seen[$url]) || $depth === 0) {
return;
}
$seen[$url] = true;
$dom = new DOMDocument('1.0');
#$dom->loadHTMLFile($url);
$anchors = $dom->getElementsByTagName('a');
foreach ($anchors as $element) {
$href = $element->getAttribute('href');
if (0 !== strpos($href, 'http')) {
// build the URLs to the same standard - with http:// etc
$path = '/' . ltrim($href, '/');
if (extension_loaded('http')) {
$href = http_build_url($url, array('path' => $path));
} else {
$parts = parse_url($url);
$href = $parts['scheme'] . '://';
if (isset($parts['user']) && isset($parts['pass'])) {
$href .= $parts['user'] . ':' . $parts['pass'] . '#';
}
$href .= $parts['host'];
if (isset($parts['port'])) {
$href .= ':' . $parts['port'];
}
$href .= $path;
}
}
crawl_page($href, $depth - 1);
}
// pull out the actual page name without any parent dirs
$pos = strrpos($url, '/');
$slug = $pos === false ? "root" : substr($url, $pos + 1);
echo "slug:" . $slug . "<br>";
}