I wrote some codes in PHP to extract content of some elements from other websites. These elements are addressed by XPath. These codes worked for one website successfully but failed for the other one. Therefore, I am sure that not the whole code is incorrect.
by the way: I extracted the element's XPath address by using 'Inspect Element' in Firefox and Right Click on the element and choosing 'Copy XPath'.
What is wrong for the second website?
thanks
Here is the code:
//MyCode.PHP
<html>
<head>
<title>This is the title</title>
</head>
<body>
<?php
class EmDIV
{
public $url="";
public $content="";
public $name="";
public $query="";
public function EmDIV($CdivName,$Curl,$CQuery)
{
$this->name=$CdivName;
$this->url=$Curl;
$this->query=$CQuery;
$html = new DOMDocument();
#$html->loadHtmlFile($this->url);
$bodies = $html->getElementsByTagName('body');
assert($bodies->length === 1);
$body = $bodies->item(0);
$xpath = new DOMXPath( $html );
$nodelist = $xpath->query($this->query);
//echo #$body->saveHTML();
if($nodelist->length==1)
{
$this->content=$nodelist->item(0)->textContent;
//sanitizing
//$this->content=Jsoup.clean($this->content, Whitelist.basic());
}
//echo $nodelist->item(0)->nodeName;
foreach ($nodelist as $node)
echo $node->getNodePath()."\n";
}
}
$emdiv=array(
//new EmDIV('parsmalaysia','http://www.parsmalaysia.com/exchange.html','/html/body/div/div[5]/div/div/div/div/div/div/div/div/div/div/table/tbody/tr/td[4]/text()'),
new EmDIV('atlas-exchange','http://atlas-exchange.com/','/html/body/div/div/table/tr/td/table/tr/td/div/div[10]/div/div/div/table/tr[3]/td[2]/text()'),
new EmDIV('usunmalaysia','http://www.usunmalaysia.com/Home.aspx','/html/body/form/table/tbody/tr[3]/td/table/tbody/tr/td/table/tbody/tr/td/table/tbody/tr[4]/td/table/tbody/tr/td/table/tbody/tr/td[2]/div/table/tbody/tr[2]/td/table/tbody/tr/td/table/tbody/tr[2]/td/table/tbody/tr[3]/td/table/tbody/tr[4]/td[2]/text()'),
);
?>
<table border="1">
<tr>
<td>Site</td>
<td>RM Price</td>
</tr>
<?php
foreach ($emdiv as $ed)
{
echo "<tr>";
echo "<td>".$ed->name."</td>";
echo "<td>".$ed->content."</td>";
echo "</tr>";
}
?>
</table>
</body>
</html>
In the second page there are no tbody elements. You have to remove all tbody elementes from the path. Firefox show the tbody elements because they are part of the internal structure of the tables, but they are not in the markup you are retrieving.
Related
I have a function that get all <h2> using DOMDocument,
Now I want to check if there is any HTML tag between <h2>[here]</h2>, don't get the <h2> and skip to next.
My Code:
foreach ($DOM->getElementsByTagName('*') as $element) {
if ($element->tagName == 'h2') {
$h = $element->textContent;
}
}
I think the easiest thing is to just reuse getElementsByTagName("*") on the element and count how many items are found.
$html = <<<EOT
<html><body><h2>Hello</h2> <h2>World</h2><h2><strong>!</strong></h2></body></html>
EOT;
$dom = new DOMDocument();
$dom->loadHTML($html);
foreach($dom->getElementsByTagName('h2') as $h2) {
if(!count($h2->getElementsByTagName('*'))){
var_dump($h2->textContent);
}
}
Demo here: https://3v4l.org/dI1e4
I have a simple HTML construct:
<table>
<tr>
<td class="myStyle">
Name of URL a
</td>
</tr>
<tr>
<td class="myStyle">
Name of URL b
</td>
</tr>
</table>
Now I want to find out with PHP DOM how to get this URL and perhaps the name.
while($table = $tables->item($i++))
{
$class_node = $table->attributes->getNamedItem('class');
if($class_node)
{
if ($table->attributes->getNamedItem('class')->value == "myStyle") {
$links = $tables->item($i)->getElementsByTagName('a');
foreach ($links as $link) {
echo "<br>" . $link->nodeName;
}
//echo "Class is : " . $table->attributes->getNamedItem('class')->value . PHP_EOL;
}
}
}
So far I can print out every row that has the class "myStyle". But I'm not able to get Access to any value or href from there.
I know XPath is much more convenient but I want to try it first with DOM.
In XPath I know my Position when traversing through the DOM. But here with DOM I guess I'm not at that position in my Loop I want.
Proceed like this..
$dom = new DOMDocument;
$dom->loadHTML($html);
foreach ($dom->getElementsByTagName('td') as $tag) {
if ($tag->getAttribute('class') === 'myStyle') {
foreach ($tag->getElementsByTagName('a') as $atag) {
echo $atag->getAttribute('href');
}
}
}
OUTPUT :
URLa
URLb
Demo
I have code trying to extract the Event SKU from the Robot Events Page, here is an example. The code that I am using dosn't find any of the SKU on the page. The SKU is on line 411, with a div of the class "product-sku". My code doesn't event find the Div on the page and just downloads all the events. Here is my code:
<?php
require('simple_html_dom.php');
$html = new simple_html_dom();
if(!$events)
{
echo mysqli_error($con);
}
while($event = mysqli_fetch_row($events))
{
$htmldown = file_get_html($event[4]);
$html->load($htmldown);
echo "Downloaded";
foreach ($html->find('div[class=product-sku]') as $row) {
$sku = $row->plaintext;
echo $sku;
}
}
?>
Can anyone help me fix my code?
This code is used DOMDocument php class. It works successfully for below sample HTML. Please try this code.
// new dom object
$dom = new DOMDocument();
// HTML string
$html_string = '<html>
<body>
<div class="product-sku1" name="div_name">The this the div content product-sku</div>
<div class="product-sku2" name="div_name">The this the div content product-sku</div>
<div class="product-sku" name="div_name">The this the div content product-sku</div>
</body>
</html>';
//load the html
$html = $dom->loadHTML($html_string);
//discard white space
$dom->preserveWhiteSpace = TRUE;
//the table by its tag name
$divs = $dom->getElementsByTagName('div');
// loop over the all DIVs
foreach ($divs as $div) {
if ($div->hasAttributes()) {
foreach ($div->attributes as $attribute){
if($attribute->name === 'class' && $attribute->value == 'product-sku'){
// Peri DIV class name and content
echo 'DIV Class Name: '.$attribute->value.PHP_EOL;
echo 'DIV Content: '.$div->nodeValue.PHP_EOL;
}
}
}
}
I would use a regex (regular expression) to accomplish pulling skus out.
The regex:
preg_match('~<div class="product-sku"><b>Event Code:</b>(.*?)</div>~',$html,$matches);
See php regex docs.
New code:
<?php
if(!$events)
{
echo mysqli_error($con);
}
while($event = mysqli_fetch_row($events))
{
$htmldown = curl_init($event[4]);
curl_setopt($htmldown, CURLOPT_RETURNTRANSFER, true);
$html=curl_exec($htmldown);
curl_close($htmldown)
echo "Downloaded";
preg_match('~<div class="product-sku"><b>Event Code:</b>(.*?)</div>~',$html,$matches);
foreach ($matches as $row) {
echo $row;
}
}
?>
And actually in this case (using that webpage) being that there is only one sku...
instead of:
foreach ($matches as $row) {
echo $row;
}
You could just use: echo $matches[1]; (The reason for array index 1 is because the whole regex pattern plus the sku will be in $matches[0] but just the subgroup containing the sku is in $matches[1].)
try to use
require('simple_html_dom.php');
$html = new simple_html_dom();
if(!$events)
{
echo mysqli_error($con);
}
while($event = mysqli_fetch_row($events))
{
$htmldown = str_get_html($event[4]);
echo "Downloaded";
foreach ($htmldown->find('div[class=product-sku]') as $row) {
$sku = $row->plaintext;
echo $sku;
}
}
and if class "product-sku" is only for div's then you can use
$htmldown->find('.product-sku')
how to get the header value like h1 or h2 which lies inside a div having some class name using simple html parser dom?
ex: - <html>
<body>
<div class="somename">
<h1>MyText</h1>
</div>
</body>
</html>
See the Document Object Model
$doc = new DOMDocument();
$doc->loadHTML('<html> <body> <div class="somename"> <h1>MyText</h1> </div> </body> </html>');
$els = $doc->getElementsByTagName('h1');
foreach ($els as $el) {
echo $el->nodeValue;
}
You can use xPath to locate h1 and then remove them by looping like this:
$doc = ...; // your DOM document
$xPath = new DOMXpath($doc);
$elements = $xpath->query("*[#class='somename']/h1");
if( !is_null( $elements)){
foreach ($elements as $element){
echo $element->nodeValue;
$element->parentNode->removeChild($element); //you may also delete elements
}
}
NOTE: I've written the code out of my head, please check documentation and examples.
I'm new to DOM parsing in PHP:
I have a HTML file that I'm trying to parse. It has a bunch of DIVs like this:
<div id="interestingbox">
<div id="interestingdetails" class="txtnormal">
<div>Content1</div>
<div>Content2</div>
</div>
</div>
<div id="interestingbox">
......
I'm trying to get the contents of the many div boxes using php.
How can I use the DOM parser to do this?
Thanks!
First i have to tell you that you can't use the same id on two different divs; there are classes for that point. Every element should have an unique id.
Code to get the contents of the div with id="interestingbox"
$html = '
<html>
<head></head>
<body>
<div id="interestingbox">
<div id="interestingdetails" class="txtnormal">
<div>Content1</div>
<div>Content2</div>
</div>
</div>
<div id="interestingbox2">a link</div>
</body>
</html>';
$dom_document = new DOMDocument();
$dom_document->loadHTML($html);
//use DOMXpath to navigate the html with the DOM
$dom_xpath = new DOMXpath($dom_document);
// if you want to get the div with id=interestingbox
$elements = $dom_xpath->query("*/div[#id='interestingbox']");
if (!is_null($elements)) {
foreach ($elements as $element) {
echo "\n[". $element->nodeName. "]";
$nodes = $element->childNodes;
foreach ($nodes as $node) {
echo $node->nodeValue. "\n";
}
}
}
//OUTPUT
[div] {
Content1
Content2
}
Example with classes:
$html = '
<html>
<head></head>
<body>
<div class="interestingbox">
<div id="interestingdetails" class="txtnormal">
<div>Content1</div>
<div>Content2</div>
</div>
</div>
<div class="interestingbox">a link</div>
</body>
</html>';
//the same as before.. just change the xpath
[...]
$elements = $dom_xpath->query("*/div[#class='interestingbox']");
[...]
//OUTPUT
[div] {
Content1
Content2
}
[div] {
a link
}
Refer to the DOMXPath page for more details.
I got this to work using simplehtmldom as a start:
$html = file_get_html('example.com');
foreach ($html->find('div[id=interestingbox]') as $result)
{
echo $result->innertext;
}
Very nice function from http://www.sitepoint.com/forums/showthread.php?611393-php5-need-something-like-innerHTML-instead-of-nodeValue
function innerXML($node)
{
$doc = $node->ownerDocument;
$frag = $doc->createDocumentFragment();
foreach ($node->childNodes as $child)
{
$frag->appendChild($child->cloneNode(TRUE));
}
return $doc->saveXML($frag);
}
$dom = new DOMDocument();
$dom->loadXML('
<html>
<body>
<table>
<tr>
<td id="foo">
The first bit of Data I want
<br />The second bit of Data I want
<br />The third bit of Data I want
</td>
</tr>
</table>
<body>
<html>
');
$xpath = new DOMXPath($dom);
$node = $xpath->evaluate("/html/body//td[#id='foo' ]");
$dataString = innerXML($node->item(0));
$dataArr = explode("<br />", $dataString);
$dataUno = $dataArr[0];
$dataDos = $dataArr[1];
$dataTres = $dataArr[2];
echo "firstdata = $nameUno<br />seconddata = $nameDos<br />thirddata = $nameTres<br />"
WebExtractor: https://github.com/knyga/webextractor
It can parse page with css, regex, xpath selectors.
Look package and tests for examples:
use WebExtractor\DataExtractor\DataExtractorFactory; use
WebExtractor\DataExtractor\DataExtractorTypes; use
WebExtractor\Client\Client;
$factory = DataExtractorFactory::getFactory(); $extractor =
$factory->createDataExtractor(DataExtractorTypes::CSS); $client = new
Client; $content =
$client->get('https://en.wikipedia.org/wiki/2014_Winter_Olympics');
$extractor->setContent($content); $h1 =
$extractor->setSelector('h1')->extract();