As I am well aware that PHPDom can solve half of my problem, I'm in need of a way (not necessarily regex) to be able to find a certain DOM element based on a given innerHTML.
say for example i got this code:
<tr>
<td class="ranking_rank" style="vertical-align:middle;">48697</td>
<td class="ranking_ign" style="vertical-align:middle;">kanineh</td>
<td class="ranking_img" style="vertical-align:middle;">
<img src="http://avatar.maplesea.com/Character/NKGEHGDLFNINKPMFLDCNNOHKHKBOHBKLGCBLABFLABHAGBPAEMDEFABJBLKJIHJAANGEKFJGELEPKMCNLKPCINEJDGAJFLKG.gif" onerror="this.src='/images/ranking/noimage.jpg'"/>
</td>
<td class="ranking_lvl" style="vertical-align:middle;">122</td>
<td class="ranking_world" style="vertical-align:middle;">
<img src="/images/ranking/Bootes.gif" onMouseover="ddrivetip('Bootes','white', 70)" onMouseout="hidetip()">
</td>
<td class="ranking_job" style="vertical-align:middle;">
<img src="/images/ranking/Warrior.gif" onMouseover="ddrivetip('Warrior','white', 70)" onMouseout="hidetip()">
</td>
<td class="ranking_fame" style="vertical-align:middle;">449</td>
</tr>
<tr>
<td class="ranking_rank" style="vertical-align:middle;">48698</td>
<td class="ranking_ign" style="vertical-align:middle;">WannaLogic</td>
<td class="ranking_img" style="vertical-align:middle;">
<img src="http://avatar.maplesea.com/Character/DOMELFGEGCGDBFCOLADBDOJLHADCIBNKEGKGINPNBEKPDDKOEEGBLMDLBGBDHGCNPGLAECAMLGKEMDKJGPODIDKCOJCMNNKN.gif" onerror="this.src='/images/ranking/noimage.jpg'"/>
</td>
<td class="ranking_lvl" style="vertical-align:middle;">122</td>
<td class="ranking_world" style="vertical-align:middle;">
<img src="/images/ranking/Aquila.gif" onMouseover="ddrivetip('Aquila','white', 70)" onMouseout="hidetip()">
</td>
<td class="ranking_job" style="vertical-align:middle;">
<img src="/images/ranking/Magician.gif" onMouseover="ddrivetip('Magician','white', 70)" onMouseout="hidetip()">
</td>
<td class="ranking_fame" style="vertical-align:middle;">56</td>
</tr>
I need to be able to get a hold of the whole row node with the td that has WannaLogic in it. that way, when I have this table row already, I can now easily traverse the nodes using PHP DOM. I'm a sucker for regular expression so I'd really much appreciate it if you can shed me some light on this.
Using regex on a DOM tree is a no-no and bound to fail when faced with malformed XML/HTML. Try this:
$xpath = new DOMXPath($doc);
$query = "//*[.='WannaLogic']";
$entries = $xpath->query($query);
foreach ($entries as $entry) {
// do whatever
}
Related
I'm trying to get a better understanding of PHP Simple HTML DOM and am kinda stuck on the following.
I am trying to retrieve information from one of my user pages by using the following code :
$dom = file_get_html('http://127.0.0.1/comments/top-commenters/');
foreach($dom->find('tr[id*=commenter]') as $result) {
print_r($result->innertext);
}
Which produces for each commenter profile ($result->innertext) the following :
<td class="Position"># 3 </td>
<td class="img" align="center">
<a href="/images/users/814ocnqlN6.jpg">
<img src="/images/users/814ocnqlN6.jpg" info="Image" border="0"/></a>
<a uid="814ocnqlN6"></td>
<td> <b>User 3.</b>
<div class="tiny">Most recent comments</div>
</td>
<td class="NumCredits"> 471 </td>
<td class="NumComments"> 5.439 </td>
<td class="PercUpVotes"> 93% </td>
Now if I would like to access within each result (same foreach loop) for example :
<td class="Position"># 3 </td>
And
<td class="NumComments"> 5.439 </td>
What would be the best way to accomplish this ?
Try:
$dom = file_get_html('http://127.0.0.1/comments/top-commenters/');
foreach($dom->find('tr[id*=commenter]') as $result) {
print_r($result->find('td.Position'));
print_r($result->find('td.NumComments'));
}
}
I have a html file that contains this table row:
<tr>
<td class="color21 right" style="font-size:12px; line-height:1.2;"> Location</td>
<td class="color21" style="font-size:12px;">10</td>
<td class="color21" style="font-size:12px;"><img src="../../icons/9.gif" alt="Type" /> </td>
<td class="color21" style="font-size:12px;">3</td>
<td class="color21" style="font-size:12px;">7</td>
<td class="color21" style="font-size:12px;"><img src="../../icons/11.gif" alt="Type" /> </td>
<td class="color21" style="font-size:12px;">3</td>
<td class="color21" style="font-size:12px;">10</td>
<td class="color21" style="font-size:12px;"><img src="../../icons/9.gif" alt="Type" /> </td>
</tr>
I'm retrieving file contents using file_get_contents.
How can I extract all TD values using preg_match, preg_match_all?
Use the DomParser to Parse the html content regex are not reliable on this cases.
$str=file_get_contents('read.txt');
$dom = new domDocument;
$dom->loadHTML($str);
$tr = $dom->getElementsByTagName('td');
foreach($tr as $td)
{
if(!empty($td->nodeValue)){
echo $td->nodeValue."\n";
}else{
$images=$td->getElementsByTagName('img');
foreach($images as $image){
echo $image->getAttribute('src')." ";
echo $image->getAttribute('alt');
}
}
Think over if you really wanna a regex to parse html
But you can use this:
<td.+?>(.+?)</td>
The first group will contain the values of <td>
I have something along the following lines in terms of HTML. I would like to extract the various contents of the table cells, however I discovered that there are some embedded divs occasionally in the cells and perhaps other oddities that I'm not sure of yet:
<p align="center">
<img src="some_image.gif" alt="Some Title">
</p>
<TABLE WIDTH=500 BORDER=1 class=textwhite ALIGN=center CELLPADDING=0 CELLSPACING=0>
<TR>
<TD colspan=4 ALIGN=center><b>Title</b></TD>
</TR>
<TR>
<TD ALIGN=center>Title</TD>
<TD ALIGN=center>date</TD>
<TD ALIGN=center>value</TD>
<TD ALIGN=center>value</TD>
</TR><TR>
<TD ALIGN=center>Title2</TD>
<TD ALIGN=center></TD>
<TD ALIGN=center><div class=redtext>----</div></TD>
<TD> </TD>
</TR><TR>
<TD ALIGN=center>Title3</TD>
<TD ALIGN=center><div class=yellowtext>value</div></TD>
<TD ALIGN=center><div class=redtext>value</div></TD>
<TD ALIGN=center>value<SUP>6</SUP></TD>
</TR><TR>
<TD ALIGN=center>Title4</TD>
<TD ALIGN=center><div class=bluetext>value</div></TD>
<TD ALIGN=center><div class=redtext>value</div></TD>
<TD> </TD>
</TR></TABLE>
<blockquote>
<p class="textstyle">
Text.
</p>
</blockquote>
My first impulse was to extract ALL element texts and just programmatically slice it up. I would watch for Title1, Title2, etc. to know when a row starts and then if a "----" is found meaning no value, just skip this row and move on. However, I realized that there is probably a better way of handling this with xpath directly.
How could this be solved with xpath so as to essentially give each cell's final child text content vs having to walk into each div if it exists? Or is there a more xpath like way to approach this?
Obviously I'm attempting to have the most flexible solution that will not be brittle if other unexpected elements crop up, even though they are unlikely.
The provided text isn't well-formed XML document, therefore XPath isn't applicable.
If you correct and covert it to a well-formed xml document as the one below, an expression like this might be useful:
/*/TABLE//TD//text()
or even:
//TABLE//TD//text()
Here is a wellformed XML document, constructed from the provided HTML:
<html>
<p align="center">
<img src="some_image.gif" alt="Some Title"/>
</p>
<TABLE WIDTH="500" BORDER="1" class="textwhite" ALIGN="center" CELLPADDING="0" CELLSPACING="0">
<TR>
<TD colspan="4" ALIGN="center">
<b>Title</b>
</TD>
</TR>
<TR>
<TD ALIGN="center">Title</TD>
<TD ALIGN="center">date</TD>
<TD ALIGN="center">value</TD>
<TD ALIGN="center">value</TD>
</TR>
<TR>
<TD ALIGN="center">Title2</TD>
<TD ALIGN="center"></TD>
<TD ALIGN="center">
<div class="redtext">----</div>
</TD>
<TD> </TD>
</TR>
<TR>
<TD ALIGN="center">Title3</TD>
<TD ALIGN="center">
<div class="yellowtext">value</div>
</TD>
<TD ALIGN="center">
<div class="redtext">value</div>
</TD>
<TD ALIGN="center">value
<SUP>6</SUP>
</TD>
</TR>
<TR>
<TD ALIGN="center">Title4</TD>
<TD ALIGN="center">
<div class="bluetext">value</div>
</TD>
<TD ALIGN="center">
<div class="redtext">value</div>
</TD>
<TD> </TD>
</TR>
</TABLE>
<blockquote>
<p class="textstyle"> Text. </p>
</blockquote>
</html>
So maybe you don't want to walk the divs, but here is my solution using lxml, which I highly recommend:
import re
from cStringIO import StringIO
from lxml import etree
def getTable(html, table_xpath, rows_xpath, cells_xpath):
"""Get a table on a webpage"""
parser = etree.HTMLParser()
# Build document tree and get table
root = etree.parse(StringIO(html), parser)
table = root.find(table_xpath)
if table == None:
print 'No table.'
return []
rows = table.findall(rows_xpath)
document = []
def cleanText(text):
"""Clean up text by replacing line breaks and tabs. """
return re.sub(r'[\r\n\t]+','',str(text).strip())
# iterate over the table rows and collect text from each cell.
for r in rows:
cells = r.findall(cells_xpath)
rowdata = []
for c in cells:
text = ''
it = c.itertext()
for i in it:
text += cleanText(i) + ' '
rowdata.append(text)
document.append(rowdata)
return document
html = """
<html><head><title></title></head><body>
<p align="center">
<img src="some_image.gif" alt="Some Title">
</p>
<TABLE WIDTH=500 BORDER=1 class=textwhite ALIGN=center CELLPADDING=0 CELLSPACING=0>
<TR>
<TD colspan=4 ALIGN=center><b>Title</b></TD>
</TR>
<TR>
<TD ALIGN=center>Title</TD>
<TD ALIGN=center>date</TD>
<TD ALIGN=center>value</TD>
<TD ALIGN=center>value</TD>
</TR><TR>
<TD ALIGN=center>Title2</TD>
<TD ALIGN=center></TD>
<TD ALIGN=center><div class=redtext>----</div></TD>
<TD> </TD>
</TR><TR>
<TD ALIGN=center>Title3</TD>
<TD ALIGN=center><div class=yellowtext>value</div></TD>
<TD ALIGN=center><div class=redtext>value</div></TD>
<TD ALIGN=center>value<SUP>6</SUP></TD>
</TR><TR>
<TD ALIGN=center>Title4</TD>
<TD ALIGN=center><div class=bluetext>value</div></TD>
<TD ALIGN=center><div class=redtext>value</div></TD>
<TD> </TD>
</TR></TABLE>
</body>
</html>
"""
tp = "//table[#width='500']"
rt = "tr"
cp = "td[#align='center']"
doc = getTable(html, tp, rt, cp)
print repr(doc)
I believe that your program is going to run into many problems as the input data is manipulated -- what if the case of 'title' changes, or there is a typo?
It's not really possible to make a rigorous solution to scraping someone else's website, as they can at no notice completely change everything. Better is normally to write tolerant and flexible code that at least tries to verify that its output is sane. In this case it's probably best to iterate over the results of '//table/tr', then inside this loop, process the td elements:
import lxml.etree
tree = lxml.etree.fromstring("<table><tr><td>test</td></tr><tr><td><div>test2</div></td></tr></table>")
stringify = lambda x : "".join(x.xpath(".//text()"))
for x in tree.xpath("//table/tr"):
print "New row"
for y in x.xpath("td"):
print stringify(y)
Output:
New row
test
New row
test2
The following code will, however, get the list you ask for:
print map(stringify, tree.xpath("//table/tr/td"))
Output:
['test', 'test2']
This will find all text elements which are at all descended from a td which is a direct descendant of a tr which is in turn a direct descendant of a table.
(Simply asking for all text() elements will create some funny bugs when run on HTML which contains "<td>Foo <b>bar</b></td>" or similar.)
below is the markup im pulling from my database table. basically i want to replace the image
<img src="http://newvision.co.ug/IM/logo_white_big.gif" width="80" style="background-color:white;padding:1px">
to
<div style='background:url(http://newvision.co.ug/IM/logo_white_big.gif) center center no-repeat;width:40px;height:40px'></div>
I dnt wanna use regular expressions just an htmlparser that ships with php
<table>
<tbody>
<tr>
<td valign="top"><a href="http://newvision.co.ug/PA/8/13/748484" target=
"_blank"><img src="http://newvision.co.ug/IM/logo_white_big.gif" width="80"
style="background-color:white;padding:1px" /></a></td>
<td valign="top">
<table>
<tbody>
<tr>
<td></td>
</tr>
<tr>
<td valign="top"><b><a target="_blank" href=
"http://newvision.co.ug/PA/8/13/748484" style="font-size:9pt">The New
Vision Online : Holland withholds sh10b over CHOGM</a></b></td>
</tr>
<tr>
<td valign="top"><a href="http://newvision.co.ug/PA/8/13/748484" style=
"font-size:8pt;color<img src="smilies/worry.gif" alt="worry" />ilver"
target="_blank">http://newvision.co.ug/PA/8/13/748484</a></td>
</tr>
<tr>
<td valign="top" style="font-size:8pt;font-weight:normal">The New Vision
is Uganda's leading daily newspaper.</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
There is no parser that ships with PHP, so use PHPQuery, a way of manipulating the DOM in a JQuery like manner instead. This will allow you to use selectors to easily swap out chunks of HTML.
I've an XML file like this:
<tr class="station">
<td class="realtime">
<span>
15:11
</span>
</td>
</tr>
<tr class="station">
<td class="clock">
15:20
</td>
</tr>
<tr class="station">
<td class="clock">
15:30
</td>
</tr>
<tr class="station">
<td class="realtime">
<span>
15:41
</span>
</td>
</tr>
and I wanna parse it with xpath in php. The xml is been updated and parsed quite often.
I always want to get the first time (in this case 15:11)
The problem is that its not sure whether the surrounding tag is a td by class "clock" or "realtime".
If there is a so surrounding realtime, then there is a span tag within. Otherwise not.
In fact, its always the first "station"-class tag in which the information is, that matters.
So is it possible to tell xpath to just evaluate within this tag?
Is there a good method for doing this in xpath?
(sry for my bad english)
In fact, its always the first
"station"-class tag in which the
information is, that matters. So is it
possible to tell xpath to just
evaluate within this tag?
With this wellformed input source:
<table>
<tr class="station">
<td class="realtime">
<span>
15:41
</span>
</td>
</tr>
<tr class="station">
<td class="clock">
15:20
</td>
</tr>
<tr class="station">
<td class="clock">
15:30
</td>
</tr>
<tr class="station">
<td class="realtime">
<span>
15:41
</span>
</td>
</tr>
</table>
This XPath expression:
/table/tr[#class='station'][1]/td
Note: Just select the element you want and use the proper DOM API method to get the string value. It doesn't matter whether there is a span element or not.
If you want to...
/table/tr[#class='station'][1]/td//text()