PHP Crawler going in Infinite Loop - php

Question

This is the whole crawler code that I am trying to build. This code is a single domain crawler. But it has a big problem, when I checked the database it was saving some of the links again and again, which creates an infinite loop. I want to solve this problem without using my database because checking each link for a presence in my database will make this crawler slow. How can I do that? + If you have any suggestions to make it faster?
<?php
include_once('ganon.php');
ini_set('display_errors', '1');
function gethost($link)
{
$link = trim($link, '/');
if (!preg_match('#^http(s)?://#', $link))
{
$link = 'http://' . $link;
}
$urlParts = parse_url($link);
$domain = preg_replace('/^www\./', '', $urlParts['host']);
return $domain;
}
function store($raw, $link)
{
$html = str_get_dom($raw);
$title = $html('title', 0)->getPlainText();
$con = #mysqli_connect('somehost', 'someuser', 'somepassword', 'somedatabase');
if (!$con)
{
echo "Error: " . mysqli_connect_error();
exit();
}
$query = "INSERT INTO `somedatabase`.`sometable` (`title`, `url`) VALUES ('$title', '$link');";
mysqli_query($con, $query);
mysqli_close($con);
echo $title."<br>";
}
function crawl_save_crawl($target)
{
$curl = curl_init($target);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
$result = curl_exec($curl);
if(curl_errno($curl))
{
echo 'Curl error: ' . curl_error($curl);
}
curl_close($curl);
$dom = str_get_dom($result);
foreach($dom('a') as $element)
{
$href = $element->href;
if (0 !== strpos($href, 'http'))
{
$path = '/' . ltrim($href, '/');
if (extension_loaded('http'))
{
$href = http_build_url("http://www.".gethost($target), array('path' => $path));
}
else
{
$parts = parse_url("http://www.".gethost($target));
$href = $parts['scheme'] . '://';
if (isset($parts['user']) && isset($parts['pass']))
{
$href .= $parts['user'] . ':' . $parts['pass'] . '#';
}
$href .= $parts['host'];
if (isset($parts['port']))
{
$href .= ':' . $parts['port'];
}
$href .= dirname($parts['path'], 1).$path;
}
}
if (gethost($target) == gethost($href))
{
crawl_save_crawl($href);
}
}
store($result, $target);
}
$url=$_GET['u'];
crawl_save_crawl($url);
?>

Answer 1

In your crawl_save_crawl() function, you could store the links you've already visited and therefore stop the code going back to them. Using a static variable isn't ideal, but in such a limited piece of code it serves the purpose it was intended for (to hold it's value across calls).
This doesn't stop it searching things other pages have searched, but stops it looping in itself.
function crawl_save_crawl($target)
{
static $alreadyDone = null;
if ( $alreadyDone == null ) {
$alreadyDone = [$target];
}
In the first loop, this will add the current reference in as this would previously have caused it to be missing.
Then before you call the same routing, you can test if it's already been checked...
$visit = trim(str_replace(["http://","https://"], "", $href), '/');
if (in_array($visit, $alreadyDone) === false &&
gethost($target) == gethost($href))
{
$alreadyDone[] = $visit;
crawl_save_crawl($href);
}
This may still look as though it's visiting the same page, but as your logic sometimes creates a href with 'www.' at the start, this may be different than without it. So when ccrawling stackoverflow, this means there is stackoverflow.com and www.stackoverflow.com.