Scraping within a table using PHP DOM - php

Question

So I have been experimenting with the PHP Simple HTML Parser and the built-in PHP DOM parser for PHP 5 to scrape a website.
We'll take this as an example: http://www.ammunitiondepot.com/12-Gauge-Shotgun-Ammo-s/1922.htm
I am trying to grab all of the product images within the v65-productDisplay table. I am able to grab all of the images on the page, but am having difficulty trying to grab only the images within the table.
This is the code I am using to grab all of the images:
$html = file_get_contents('http://www.ammunitiondepot.com/12-Gauge-Shotgun-Ammo-s/1922.htm');
$dom = new domDocument;
$dom->loadHTML($html);
$dom->preserveWhiteSpace = false;
$images = $dom->getElementsByTagName('img');
foreach ($images as $image) {
echo $image->getAttribute('src');
}

Answer 1

The problem you get all <img> src-attributes is basically because you run
$images = $dom->getElementsByTagName('img');
That says you want to get all <img> elements from the whole document. But actually you want to get only those images that are inside a specific table. At least you right now think so, but let's do this straight for the moment. The table you're looking for is the 13th table in document order. So what you do now to fix your problem is you first get the <table> and then you get all <img> elements from it:
$tables = $dom->getElementsByTagName('table');
$table = $tables->item(12); # 13th table in the document
$images = $table->getElementsByTagName('img');
This will then already give you the images you ask for in your question (excerpt of src attributes):
//cdn3.volusion.com/sfvhn.cpdkd/v/vspfiles/photos/SPL12-00BK-1.jpg?1381182668
/v/vspfiles/templates/ammunition/images/clear1x1.gif
/v/vspfiles/templates/ammunition/images/clear1x1.gif
/v/vspfiles/templates/ammunition/images/clear1x1.gif
/v/vspfiles/templates/ammunition/images/clear1x1.gif
/v/vspfiles/templates/ammunition/images/buttons/btn_addtocart_small.gif
/v/vspfiles/templates/ammunition/images/clear1x1.gif
...
Obviously this leads to some further problems:
The list of images has tons of those you're not interested in. You want the .jpg files and not all those many 1-pixel-gifs or shopping-cart-buttons.
The number of the table is hardcoded. This is not very stable, it would be better lets say to look for the class="v65-productDisplay" attribute (you write it even in your question already).
The image URLs are relative to the document, so need to be resolved.
I'll show first now how to solve the first two problems.
It seems that getElementsByTagName even if useful is not that flexible for your scraping needs. And there is a better way to query elements from the document, and that is called xpath (ref). It's a query language in which you express which elements you want. So we want image src attributes from within a specific table being jpegs. The xpath query does look like this:
//table[#class="v65-productDisplay"]/tr/td/a/img/#src[contains(., ".jpg")]
This is run with the help of a DOMXPath illustrated as following:
$xpath = new DOMXPath($dom);
$srcQuery = '//table[#class="v65-productDisplay"]/tr/td/a/img/#src[contains(., ".jpg")]';
/** #var DOMAttr $src */
foreach ($xpath->query($srcQuery) as $src) {
echo $src->nodeValue, "\n";
}
This now already reduces the list greatly to what you're looking for while being less verbose:
//cdn3.volusion.com/sfvhn.cpdkd/v/vspfiles/photos/SPL12-00BK-1.jpg?1381182668
//cdn3.volusion.com/sfvhn.cpdkd/v/vspfiles/photos/GTL1275-1.jpg?1380206953
//cdn3.volusion.com/sfvhn.cpdkd/v/vspfiles/photos/SS12L8-1.jpg?1390326206
//cdn3.volusion.com/sfvhn.cpdkd/v/vspfiles/photos/LEF127RS-1.jpg?1368458526
//cdn3.volusion.com/sfvhn.cpdkd/v/vspfiles/photos/LE13300-1.jpg?1368458467
//cdn3.volusion.com/sfvhn.cpdkd/v/vspfiles/photos/ADLE13300AC-1.jpg?1393516003
...
So now only the problem is left with the cleanup of the URIs, that is resolution to the document URI (as there is no further base URI in the document) and probably cleaning up the query string. I do this with the help of Net_URL2, here the src processing alone:
/** #var DOMAttr $src */
foreach ($xpath->query($srcQuery) as $src) {
$href = $uri->resolve($src->nodeValue);
$href->setQuery(false);
echo $href, "\n";
}
And here is a full example:
<?php
/*
* #link http://stackoverflow.com/questions/24344420/scraping-within-a-table-using-php-dom
* #auhtor hakre <http://hakre.wordpress.com>
*/
# uses Net_URL2 -- http://pear.php.net/package/Net_URL2/ -- https://packagist.org/packages/pear/net_url2
require __DIR__ . '/vendor/autoload.php';
$uri = new Net_URL2('http://www.ammunitiondepot.com/12-Gauge-Shotgun-Ammo-s/1922.htm');
$cache = '12-Gauge-Shotgun-Ammo-s-1922.htm';
if (is_readable($cache)) {
$html = file_get_contents($cache);
} else {
$options = [
'http' => [
'user_agent' => "Godzilla/42.4 (Gabba Gandalf Client 7.3; C128; Z80) Lord of the Table Weed Edition (KHTML, like Gold Dust Day Gecko) Chrome/97.0.43043.0 Safari/1337.42",
'max_redirects' => 1, # do not follow redirects
]
];
$context = stream_context_create($options);
$html = file_get_contents($uri, null, $context);
file_put_contents($cache, $html);
}
$dom = new domDocument;
$dom->preserveWhiteSpace = false;
$save = libxml_use_internal_errors(true);
$dom->loadHTML($html);
libxml_use_internal_errors($save);
$dom->documentURI = $uri;
$xpath = new DOMXPath($dom);
$srcQuery = '//table[#class="v65-productDisplay"]/tr/td/a/img/#src[contains(., ".jpg")]';
/** #var DOMAttr $src */
foreach ($xpath->query($srcQuery) as $src) {
$href = $uri->resolve($src->nodeValue);
$href->setQuery(false);
echo $href, "\n";
}
And here is the HTML for future reference: http://pastebin.com/HCTTRm9E