Listing items from a e-commerce site - php

Question

I'm working on a script in php, that takes all the products URLs from an e-commerce site, for now I'm just using the function get_file_contents() and after I search for the keyword with preg_match_all() that anticipate the item url,now, my question is, can I use a more direct and efficent way to store all this link from a website and put they on my database?

Answer 1

I've recently created a crawler system for my client's project. Basically these are the steps i followed:
Project was based on PHP and it supports multiple type of document such a xml, json and html.
I've created a base product object with the properties i needed (title, image, price, link, category, source site)
For each web site, i'm using a parser which generally uses PhpDom library with DomXpath.
Basically, i find the product listing tag, and loop through the records and create a new product list object with products object inside it (step 2).
When parsing the web site finishes, i'm sending the whole list to my base action it checks if the product with that url is already exists if not it adds to db.
Also in my server, i'm running a cron where it checks all the product links and if the response returns 404 or 500 it adds a flag to product with 1. I'm also running an other cron which checks the links again with the flag 1. If it's still responding with error code, it removes the content from my database.
This is a sample parser code that i use. I hope it will help you through the process:
$content = file_get_contents($url);
libxml_use_internal_errors(true);
$oDom = new DomDocument;
$oDom->validateOnParse = false;
$res = $oDom->loadHTML($content);
libxml_clear_errors();
$oDom->preserveWhiteSpace = false;
$oXpath = new DOMXPath($oDom);
$productNode = $oXpath->query('//div[#class="ulist span4"]');
if($productNode){
$productsList = array();
foreach($productNode as $p){
$this->oProduct = new Products();
$productURL = $oXpath->query('div[#class="ures"]/div[#class="ures-cell"]/a', $p)->item(0);
$this->oProduct->url = $this->base.'/'.$productURL->getAttribute('href');
$this->oProduct->category = $categoryID;
$this->oProduct->productPeek = $peek;
$titleNode = $oXpath->query('div[#class="ubilgialan span12"]/div[#class="span12 uadi"]/a/span', $p)->item(0);
$this->oProduct->title = trim($titleNode->nodeValue);
$priceNode = $oXpath->query('div[#class="ubilgialan span12"]/div[#class="span8 ufytalan"]/div[#class="ufyt"]/span[#class="divdiscountprice"]/span', $p)->item(0);
$this->oProduct->price = trim($priceNode->nodeValue);
$imageNode = $oXpath->query('div[#class="ures"]/div[#class="ures-cell"]/a/img', $p)->item(0);
$this->oProduct->image = $this->base."/".$imageNode->getAttribute('src');
$productsList[] = $this->oProduct;
}
if(count($productsList) > 0){
return $productsList;
}
}