php dom xpath extract all links from all folders in site - php

Question

I have searched around on stackoverflow and the web and must be missing something here. I have not found exactly what I am looking for. Maybe its called something else.. I have this code below which will grab everything fine in the first folder but will not grab other items from other folders.. example it grabs everything in front of the first / but if you have a site mysite.com/folder2/ it will not grab folder2. Everything is linked. It also does travel backwards too. If you put in the longest link of the site will go all the way to the front of the site. I am not sure what I am missing any pointers would be great. The site is a joomla site that I am trying to scrap.
<?php function storelink($web,$taken) {
$query = "INSERT INTO scanned (web, taken) VALUES ('$web', '$taken')";
mysql_query($query) or die('Error, insert query failed');
}
$target_web = "mysite.com";
$userAgent = 'bobsbot(http://www.somebot.com/bot.html)';
// make the cURL request to $target_web
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL, $target_web);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 1000);
$html= curl_exec($ch);
if (!$html) {
echo "<br />cURL error number:" .curl_errno($ch);
echo "<br />cURL error:" . curl_error($ch);
exit;
}
// parse the html into a DOMDocument
$dom = new DOMDocument();
#$dom->loadHTML($html);
// grab all the on the page
$xpath = new DOMXPath($dom);
$hrefs = $xpath->evaluate("/html/body//a");
for ($i = 0; $i < $hrefs->length; $i++) {
$href = $hrefs->item($i);
$web = $href->getAttribute('href');
storeLink($web,$target_web);
echo "<br />Link saved: $web";
} ?>

Answer 1

If I understand you correctly, you want to spider a site and save all URLs. This means you need to recurse when you encounter an URL.
The function you use to start the spider is called saveLink($web, $taken). The function you call when encountering a link is storeLink($web, $target_web). Shouldn't that be saveLink($web, $target_web)?
saveLink() should be recursive and also execute the cURL request. The cURL URL should be set to the link encountered. This way, it will parse the DOM of all links encountered and follow all links in them.