PHP DOM grabbing a specific subset of information - php

The webpage in question is http://assignments.uspto.gov/assignments/q?db=pat&pub=20060030630
Now, let's just say I want to capture the Assignees in the first assignment. The relevant code there looks like
<div class="t3">Assignee:</div>
</td>
</tr>
</table>
</td><td>
<table width="100%" cellpadding="0" cellspacing="0" border="0">
<tbody valign="top">
<tr>
<td>
<table>
<tr>
<td>
<div class="p1">
LEAR CORPORATION
</div>
</td>
</tr>
<tr>
<td><span class="p1">21557 TELEGRAPH ROAD</span></td>
</tr>
<tr>
<td><span class="p1">SOUTHFIELD, MICHIGAN 48034</span></td>
</tr>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
I could I suppose use xpath and grab everything out of spans with class p1, except that thing is used all throughout the page for basically everything, same for the div class that lear corporation is in.
So is there a way for me to just read "Assignees" and then grab just the information relevant to it?
I figure if I can understand how to do that, then I can extrapolate from that and figure out how to grab any specific data on the page that I want, i.e. grabbing the conveyance data on any particular assignment.
But if say, I were just to grab all the data on the page (reel/frame, conveyance, assignors, assignee, correspondent for every assignment, and the header information about the patent itself), might that be easier to do than trying to grab each individual piece of information?

There is no clear way to do it since we have no designation in the DOM where this information is.. It's very arbitrary.
I would recommend using some math to figure out the pattern of where in the DOM the Assignee resides.
For example, we know that for every class of p1, the assignee value is position 16, and a new Assignment occurs every 23rd position. Using a loop you could figure it out.
This should get you started at the very least.
$Site = file_get_contents('http://assignments.uspto.gov/assignments/q?db=pat&pub=20060030630');
$Dom = new DomDocument();
$Dom->loadHTML($Site);
$Finder = new DomXPath($Dom);
$Nodes = $Finder->query("//*[contains(concat(' ', normalize-space(#class), ' '), ' p1 ')]");
$position = 0;
foreach($Nodes as $node) {
if(($position % 16) == 0 && $position > 0) {
var_dump($node->nodeValue);
break;
}
$position++;
}

Related

How do you web scrape nested table using PHP?

Backstory
First off, I have used PHP to read email and extract out the HTML attachment which later on I stored in a PHP variable.
Now the problem I am facing is that I am trying to extract out information from the HTML in a nested table and hoping to convert to some sort of array so that I can store in SQL.
May I know if there are solutions for this? As I have tried finding but to no avail.
Example
<html>
<table>
<tr>
<td>
<table>
<tr>
<td></td>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td>
<table>
<tr>
<td>
<p>hi</p>
</td>
</tr>
</table>
</td>
</tr>
</table>
</html>
I want to locate the nearest table tag where "hi" is located so that I can get all the information from that table
Stuff I have tried
I have tried using simple HTML DOM but I guess the HTML file that I tried scraping was too large that it causes memory issues.
include('./simple_html_dom.php');
/* Parse the HTML, stored as a string in $woString */ <br>
$html = str_get_html($worksOrder);
/* Locate the ultimate grandfather table id that wraps all his generation */<br>
$messytables = $html->find('table');
print_r($messytables);
Rather than using simple HTML DOM, this uses DOMDocument and XPath to find the elements.
This draws on the answer XPath to find nearest ancestor element that contains an element that has an attribute with a certain value to locate the <table> tags that enclose the <p> tags which have hi in it. As there are a few levels of enclosing <table> tags, it then uses last() (from XSLT getting last element) to find the innermost enclosing <table>...
libxml_use_internal_errors(true);
$doc = new DOMDOcument();
$doc->loadHTML( $worksOrder );
$xp = new DOMXPath($doc);
$table = $xp->query('(//ancestor::table[descendant::p="hi"])[last()]');
echo $doc->saveHTML($table[0]);
The last line is just to display the data, you can just start with $table[0] and fetch the data as needed.
This outputs with your test data...
<table><tr>
<td>
<p>hi</p>
</td>
</tr></table>

How to get data from other site

I'm making a website and it'll have a user page on which will be user score. I'm taking that score from the other website but the problem is how to do that: how to get text from any element on any website?
I wanted to try this but I don't know how to select that th from which I need text because the whole table hasn't any IDs, only a class. You can see in the code, I want text from the place where is written: I NEED THIS.
Thanks.
<table class="profile" style="padding:0px; margin:0px;">
<tr>
<td valign="top" style="padding:0px; margin:0px;">
<h3>TEXT</h3>
</td>
<td valign="bottom" align="right">
TEXT</br>
TEXT</br>
</td>
</tr>
<tr>
<td valign="top">
<table class="score_stats">
<tr><th>TEXT</th><th> **I NEED THIS** </th></tr>
<tr><td>TEXT</td><td>NUMBER</td></tr>
<tr><td>TEXT</td><td>NUMBER</td></tr>
<tr><td>TEXT</td><td>NUMBER</td></tr>
<tr><td>TEXT</td><td>NUMBER</td></tr>
<tr><td>TEXT</td><td>NUMBER</td></tr>
</table>
</td>
</tr>
</table>
Get the html code of the page , in string or array
Many choices here >>How do I get the HTML code of a web page in PHP?
2.Split those strings or things you need with some simple code , I know you could find it by yourself
Hope this helps
Poom
After the whole day of trying I made this:
function getPlayerScore($url){
libxml_use_internal_errors( true );
$dom = new DOMDocument();
$dom->loadHTMLFile( $url );
$finder = new DOMXpath( $dom );
$nodes = $finder->query( "//*[#class='score_stats']/tr" )->item(0)->nodeValue;
$nodes = str_split($nodes, 11);
return $nodes[1];
}
getPlayerScore($user['playercp'])
...and it's working.

How to use DOMDocument to get child elements?

I am trying to get the text of child elements using the PHP DOM.
Specifically, I am trying to get only the first <a> tag within every <tr>.
The HTML is like this...
<table>
<tbody>
<tr>
<td>
1st Link
</td>
<td>
2nd Link
</td>
<td>
3rd Link
</td>
</tr>
<tr>
<td>
1st Link
</td>
<td>
2nd Link
</td>
<td>
3rd Link
</td>
</tr>
</tbody>
</table>
My sad attempt at it involved using foreach() loops, but would only return Array() when doing a print_r() on the $aVal.
$dom = new DOMDocument();
libxml_use_internal_errors(true);
$dom->loadHTML(returnURLData($url));
libxml_use_internal_errors(false);
$tables = $dom->getElementsByTagName('table');
$aVal = array();
foreach ($tables as $table) {
foreach ($table as $tr){
$trVal = $tr->getElementsByTagName('tr');
foreach ($trVal as $td){
$tdVal = $td->getElementsByTagName('td');
foreach($tdVal as $a){
$aVal[] = $a->getElementsByTagName('a')->nodeValue;
}
}
}
}
Am I on the right track or am I completely off?
Put this code in test.php
require 'simple_html_dom.php';
$html = file_get_html('test1.php');
foreach($html->find('table tr') as $element)
{
foreach($element->find('a',0) as $element)
{
echo $element->plaintext;
}
}
and put your html code in test1.php
<table>
<tbody>
<tr>
<td>
1st Link
</td>
<td>
2nd Link
</td>
<td>
3rd Link
</td>
</tr>
<tr>
<td>
1st Link
</td>
<td>
2nd Link
</td>
<td>
3rd Link
</td>
</tr>
</tbody>
</table>
I am pretty sure I am late, but better way should be to iterate through all "tr" with getElementByTagName and then while iterating through each node in nodelist recieved use getElementByTagName"a". Now no need to iterate through nodeList point out the first element recieved by item(0). That's it! Another way can be to use xPath.
I personally don't like SimpleHtmlDom because of the loads of extra added features it uses where a small functionality is required. In case of heavy scraping also memory management issue can hold you back, its better if you yourself do DOM Analysis rather than depending thrid party application.
Just My opinion. Even I used SHD initially but later realized this.
You're not setting $trVal and $tdVal yet you're looping them ?

Simple HTML DOM: Notice->Trying to get property of non-object

I am getting an php notice when using simple html dom to scrape a website. There are 2 notices displayed and everything rendered underneath looks perfect when using the print_r function to display it.
The website table structure is as follows:
<table class=data schedTbl>
<thead>
<tr>
<th>DATA</th>
<th>DATA</th>
<th>DATA</th>
etc....
</tr>
</thead>
<tbody>
<tr>
<td>
<div class="class1">DATA</div>
<div class="class2">SAME DATA AS PREVIOUS DIV</div>
</td>
<td>DATA</td>
<td>DATA</td>
etc....
</tr>
<tr>
<td>
<div class="class1">DATA</div>
<div class="class2">SAME DATA AS PREVIOUS DIV</div>
</td>
<td>DATA</td>
<td>DATA</td>
etc....
</tr>
<tr>
<td>
<div class="class1">DATA</div>
<div class="class2">SAME DATA AS PREVIOUS DIV</div>
</td>
<td>DATA</td>
<td>DATA</td>
etc....
</tr>
etc....
</tbody>
</table>
The code below is used to find all tr in table[class=data schedTbl]. I have a tbody selector in there, but it seems to pay no attention to this selector as it still selects the tr in the thead.
include('simple_html_dom.php');
$articles = array();
getArticles('www.somesite.com');
function getArticles($page) {
global $articles;
$html = new simple_html_dom();
$html->load_file($page);
$items = $html->find('table[class=data schedTbl] tbody tr');
foreach($items as $post) {
$articles[] = array($post->children(0)->first_child(0)->plaintext,//0 -- GAME DATE
$post->children(1)->plaintext,//1 -- AWAY TEAM
$post->children(2)->plaintext);//2 -- HOME TEAM
}
}
So, I believe notices come from the tr in the thead because I am calling on the first child of the first td which only has one record. The reason for two is there is actually two tables with the same data structure in the body.
Again, I believe there are 2 ways of solving this:
1) PROBABLY THE EASIEST (fix the find selector so the TBODY works and only selects the tds within the tbodies)
2) Figure out a way to not do the first_child filter when it is not needed?
Please let me know if you would like a snapshot of the print_r($articles) output I am receiving.
Thanks in advance for any help provided!
Sincerely,
Bill C.
Just comment out line #695 in the simple_html_dom.php
if ($m[1]==='tbody') continue;
Then it should read the tbody.

DOMDocument in php

I have just started reading documentation and examples about DOM, in order to crawl and parse the document.
For example I have part of document shown below:
<div id="showContent">
<table>
<tr>
<td>
Crap
</td>
</tr>
<tr>
<td width="172" valign="top"><img height="91" border="0" width="172" class="" src="img"></td>
<td width="10"> </td>
<td valign="top"><table cellspacing="0" cellpadding="0" border="0">
<tbody><tr>
<td height="30"><a class="px11" href="link">title</a><a><br>
<span class="px10"></span>
</a></td>
</tr>
<tr>
<td><img height="1" width="580" src="crap"></td>
</tr>
<tr>
<td align="right">
<img height="16" border="0" width="65" src="/buy">
</td>
</tr>
<tr>
<td valign="top" class="px10">
<p style="width: 500px;">description.</p>
</td>
</tr>
</tbody></table></td>
</tr>
<tr>
<td>
Crap
</td>
</tr>
<tr>
<td>
Crap
</td>
</tr>
</table>
</div>
I'm trying to use the following code to get all the tr tags and analyze whether there is crap or information inside them:
$dom = new DOMDocument();
#$dom->loadHTML($html);
$xpath = new DOMXPath($dom);
$tags = $xpath->query('.//div[#id="showContent"]');
foreach ($tags as $tag) {
$string="";
$string=trim($tag->nodeValue);
if(strlen($string)>3) {
echo $string;
echo '<br>';
}
}
However I'm getting just stripped string without the tags, for example:
Crap
Crap
Title
Description
But I would like to get:
<tr>
<td>Crap</td>
</tr>
<tr>
title
</tr>
How to keep html nodes (tags)?
If you want to work with DOM you have to understand the concept. Everything in a DOM Document, including the DOMDocument, is a Node.
The DOMDocument is a hierarchical tree structure of nodes. It starts with a root node. That root node can have child nodes and all these child nodes can have child nodes on their own. Basically everything in a DOMDocument is a node type of some sort, be it elements, attributes or text content.
HTML Legend:
/ \ UPPERCASE = DOMElement
HEAD BODY lowercase = DOMAttr
/ \ "Quoted" = DOMText
TITLE DIV - class - "header"
| \
"The Title" H1
|
"Welcome to Nodeville"
The diagram above shows a DOMDocument with some nodes. There is a root element (HTML) with two children (HEAD and BODY). The connecting lines are called axes. If you follow down the axis to the TITLE element, you will see that it has one DOMText leaf. This is important because it illustrates an often overlooked thing:
<title>The Title</title>
is not one, but two nodes. A DOMElement with a DOMText child. Likewise, this
<div class="header">
is really three nodes: the DOMElement with a DOMAttr holding a DOMText. Because all these inherit their properties and methods from DOMNode, it is essential to familiarize yourself with the DOMNode class.
In practise, this means the DIV you fetched is linked to all the other nodes in the document. You could go all the way to the root element or down to the leaves at any time. It's all there. You just have to query or traverse the document for the wanted information.
Whether you do that by iterating the childNodes of the DIV or use getElementByTagName() or XPath is up to you. You just have to understand that you are not working with raw HTML, but with nodes representing that entire HTML document.
If you need help with extracting specific information from the document, you need to clarify what information you want to fetch from it. For instance, you could ask how to fetch all the links from the table and then we could answer something like:
$div = $dom->getElementById('showContent');
foreach ($div->getElementsByTagName('a') as $link)
{
echo $dom->saveXML($link);
}
But unless you are more specific, we can only guess which nodes might be relevant.
If you need more examples and code snippets on how to work with DOM browse through my previous answers to related questions:
https://stackoverflow.com/search?q=user%3A208809+DOM
By now, there should be a snippet for every basic to medium UseCase you might have with DOM.
To create a parser you can use htmlDOM.
It is very simple easy to use DOM parser written in php. By using it you can easily fetch the contents of div tag.
For example, find all div tags which have attribute id with a value of text.
$ret = $html->find('div[id=text]');

Categories