How to make Web Crawler to crawl pages in background? - php

Question

i am currently developing a web crawler in PHP and it still is a simple one but what i want to know is how can i make my crawler to crawl pages in background and not use my bandwidth, do i have to use some cron jobs and i want it to automatically store the data in database.
Here what i have done:
<?php
$conn = mysqli_connect("localhost","root","","crawler") or die(mysqli_error());
ini_set('max_execution_time', 4000);
$to_crawl = "http://hootpile.com";
$c = array();
function get_links($url){
global $c;
$input = file_get_contents($url);
$regexp = "<a\s[^>]*href=(\"??)([^\" >]*?)\\1[^>]*>(.*)<\/a>";
preg_match_all("/$regexp/siU", $input, $matches);
$base_url = parse_url($url, PHP_URL_HOST);
$l = $matches[2];
foreach($l as $link) {
if(strpos($link, "#")) {
$link = substr($link,0, strpos($link, "#"));
}
if(substr($link,0,1) == ".") {
$link = substr($link, 1);
}
if(substr($link,0,7)=="http://") {
$link = $link;
}
else if(substr($link,0,8) =="https://") {
$link = $link;
}
else if(substr($link,0,2) =="//") {
$link = substr($link, 2);
}
else if(substr($link,0,2) =="#") {
$link = $url;
}
else if(substr($link,0,2) =="mailto:") {
$link = "[".$link."]";
}
else {
if(substr($link,0,1) != "/") {
$link = $base_url."/".$link;
}
else {
$link = $base_url.$link;
}
}
if(substr($link, 0, 7)=="http://" && substr($link, 0, 8)!="https://" && substr($link, 0, 1)=="[") {
if(substr($url, 0, 8) == "https://") {
$link = "https://".$link;
}
else {
$link = "http://".$link;
}
}
//echo $link."<br />";
if(!in_array($link,$c)) {
array_push($c,$link);
}
}
}
get_links($to_crawl);
foreach ($c as $page) {
get_links($page);
}
foreach ($c as $page) {
$query = mysqli_query($conn,"INSERT INTO LINKS VALUES('','$page')");
echo $page."<br />";
}
?>

Answer 1

You can use SimpleHTML Dom, But crawling/scraping depend on the web page structure. How many data you want to store, May be you can't found same data and structure on different websites. In case you should make some common program to fetch data from scraped data.

Answer 2

You can use ReactPHP, since it allows you to easily spawn a process that keeps running.
You also can write a hashbang at the very beggining of the file:
#/usr/bin/php
give execution permissions to the file:
chmod a+x your_script_path.php
and execute it with cron or with nohup. If you want to daemonize it, then there is a little more work.

Answer 3

I think you should not use PHP for crawler/scraper because it's not designed for long-running tasks. It will cause memory usage problem, use Python instead (I use Python + BeautifulSoup + urllib for scraper).
Besides you should use crontab and nohup to schedule background jobs.