What I've been trying to do recently is to extract listing information from a given html file,
For example, I have an html page that has a list of many companys, with their phone number, address, etc'
Each company is in it's own table, every table started like that: <table border="0">
I tried to use PHP to get all of the information, and use it later, like put it in a txt file, or just import into a database.
I assume that the way to achieve my goal is by using regex, which is one of the things that I really have problems with in php,
I would appreciate if you guys could help me here.
(I only need to know what to look for, or atleast something that could help me a little, not a complete code or anything like that)
Thanks in advance!!
I recommend taking a look at the PHP DOMDocument and parsing the file using an actual HTML parser, not regex.
There are some very straight-forward ways of getting tables, such as the GetElementsByTagName method.
<?php
$htmlCode = /* html code here */
// create a new HTML parser
// http://php.net/manual/en/class.domdocument.php
$dom = new DOMDocument();
// Load the HTML in to the parser
// http://www.php.net/manual/en/domdocument.loadhtml.php
$dom->LoadHTML($htmlCode);
// Locate all the tables within the document
// http://www.php.net/manual/en/domdocument.getelementsbytagname.php
$tables = $dom->GetElementsByTagName('table');
// iterate over all the tables
$t = 0;
while ($table = $tables->item($t++))
{
// you can now work with $table and find children within, check for
// specific classes applied--look for anything that would flag this
// as the type of table you'd like to parse and work with--then begin
// grabbing information from within it and treating it as a DOMElement
// http://www.php.net/manual/en/class.domelement.php
}
If You're familiar with jQuery (and even if You're not as it's command are simple enough) I recommend this PHP counterpart: http://code.google.com/p/phpquery/
If your HTML is valid XML, as in XHTML, then you could parse it using SimpleXML
Related
I need to download results from a website using a for loop to compile them.
(Note that it's an ASP request which displays a webpage with these parameters)
I wrote the following code to get me this:
<?php
for ($i=10; $i<500; $i++) {
$m = $i*10;
$dl = $query;
$text = file_get_contents($dl);
$doc = new DOMDocument('1.0');
$doc->loadHTML($text);
$aObj = $doc->find('Academic');
if (count($aObj) > 0)
{
echo "<h4>Found</h4>";
//Don't download this
}
else
{
echo "<h4>Not found</h4>";
//Download this
}
}
?>
But it returns several errors. Apparently it can't copy the ASPX file to the HTML DOM. How do I go about doing this? Also, how can I download/save the pages where the string 'Download' is not found?
I also think my method of finding 'Download' in the document is not working. What is the correct way to do this?
The website you're attempting to parse contains a lot of errors, therefore you wont be able to use the standard DOMDocument object. You can attempt to use a library such as SimpleHTMLDOM (http://simplehtmldom.sourceforge.net/) or phpQuery (https://code.google.com/p/phpquery/) and hope that those are good enough to parse the malformed document.
In case you just need some information it might be easier to use regular expressions and preg_match_all (http://www.php.net/manual/en/function.preg-match-all.php) to find every occurrence of 'Academic' for example.
Note, usually it is not very advisable to use regular expression when working with structured documents such as HTML since you wont be able to take advantage of the structure, but since those documents seem to contain over 300 errors and differ from each other it might be the only way.
I'm trying to insert THE PHP CODE between the <div id="container"> THE PHP CODE HERE </div> . I'm using the http://simplehtmldom.sourceforge.net/ in this Php Html Parser.
I've got this so far, how to find a div having an id called container:
// Find all which attribute id=foo
$ret = $html->find('div[id=foo]');
Then after finding the foo, what's next?
$html->find('div[id=foo]')->innertext = 'bar';
Should do the job but you might be better off modifying the DOM with JS?
Why not modify the source Wordpress template instead of trying to hack in the text later? That PHP snippet could potentially produce a huge chunk of text, and you can't insert it verbatim into the DOM - you can only insert its output (which you'd have to capture with ob_start()/ob_get_clean() anywyas).
hi there i have a question and need help on this from you guys .
I have a database created for a game called gamesleaderboard and the fields are id, player_name, score, leveltime. and my task is after getting the score, i have to insert it to a database and sort the dbase accordingly.
after sorting, the code will return an xml in the following structure:
Ahmad100080
Basel95090
Samer920100
Seyd900110
Ahmad100080
Basel95090
Samer920100
Seyd900110
Ahmad100080
Basel95090
plz tell me the necessary details how to do this thankyou.
In most simplistic terms there is nothing really special to do here, you can output XML in exactly the same way you would output HTML in PHP, this is a simple example
You can also use the DOMDocument class (or SimpleXML) to output XML, this is a bit more complex but is better practice. For an example of creating XML with DOMDocument using data from MySQL read more here
Don't forget to send the correct Content-Type header before outputting the XML document tho.
If you are very new to XML I would highly recommend SimpleXML as it will be enough for most of needs. Creating XML using "echo" and strings is not only a dangeous but also a very bad programming technique.
Using SimpleXML you can easily add new nodes, adding child nodes and attributes to them. If you can getting started reading the PHP docs, just search for a SimpleXML tutorial on google. Or ask your questions right here.
$query=mysql_query("Select * from gamesleaderboard ");
$number=mysql_num_rows($query);
if ($query==0)
{
echo "0 rows Affected";
}
$doc= new DOMDocument();
$doc->formatOutput=true;
$root= $doc->createElement("Games");
$doc->appendChild($root);
for ($i=0; $i<$number; $i++){
$row=mysql_fetch_array($qex);
$node=$doc->createElement("user");
$pn=$doc->createElement("player_name");
$pn->appendChild($doc->createTextNode($row["player_name"]));
$node->appendChild($pn);
$sc=$doc->createElement("score");
$sc->appendChild($doc->createTextNode($row["score"]));
$node->appendChild($sc);
$root->appendChild($node);
}
echo $doc->saveXML();
This will display the answer exactly you want. I just tested it. I was probably in highschool when you asked this question here. Anyway it'll help someone else.
This question already has answers here:
How to insert HTML to PHP DOMNode?
(5 answers)
Closed 7 years ago.
I am using PHP's DOM object to create HTML pages for my website. This works great for my head, however since I will be entering a lot of HTML into the body (not via DOM), I would think I would need to use DOM->createElement($bodyHTML) to add my HTML from my site to the DOM object.
However DOM->createElement seems to parse all HTML entities so my end result ended up displaying the HTML on the page and not the actual renders HTML.
I am currently using a hack to get this to work,
$body = $this->DOM
->createComment('DOM Glitch--><body>'.$bodyHTML."</body><!--Woot");
Which puts all my site code in a comment, which I bypass athe comment and manually add the <body> tags.
Currently this method works, but I believe there should be a more proper way of doing this. Ideally something like DOM->createElement() that will not parse any of the string.
I also tried using DOM->createDocumentFragment() However it does not like some of the string so it would error and not work (Along with take up extra CPU power to re-parse the body's HTML).
So, my question is, is there a better way of doing this other than using DOM->createComment()?
You use the DOMDocumentFragment objec to insert arbitrary HTML chunks into another document.
$dom = new DOMDocument();
#$dom->loadHTML($some_html_document); // # to suppress a bajillion parse errors
$frag = $dom->createDocumentFragment(); // create fragment
$frag->appendXML($some_other_html_snippet); // insert arbitary html into the fragment
$node = // some operations to find whatever node you want to insert the fragment into
$node->appendChild($frag); // stuff the fragment into the original tree
I FOUND THE SOLUTION but it's not a pure php solution, but works very well. A little hack for everybody who lost countless hours, like me, to fix this
$dom = new DomDocument;
// main object
$object = $dom->createElement('div');
// html attribute
$attr = $dom->createAttribute('value');
// ugly html string
$attr->value = "<div> this is a really html string ©</div><i></i> with all the © that XML hates!";
$object->appendChild($attr);
// jquery fix (or javascript as well)
$('div').html($(this).attr('value')); // and it works!
$('div').removeAttr('value'); // to clean-up
loadHTML works just fine.
<?php
$dom = new DOMDocument();
$dom->loadHTML("<font color='red'>Hey there mrlanrat!</font>");
echo $dom->saveHTML();
?>
which outputs Hey there mrlanrat! in red.
or
<?php
$dom = new DOMDocument();
$bodyHTML = "here is the body, a nice body I might add";
$dom->loadHTML("<body> " . $bodyHTML . " </body>");
// this would even work as well.
// $bodyHTML = "<body>here is the body, a nice body I might add</body>";
// $dom->loadHTML($bodyHTML);
echo $dom->saveHTML();
?>
Which outputs:
here is the body, a nice body I might add and inside of your HTML source code, its wrapped inside body tags.
I spent a lot of time working on Anthony Forloney's answer, But I cannot seem to get the html to append to the body without it erroring.
#Mark B: I have tried doing that, but as I said in the comments, it errored on my html.
I forgot to add the below, my solution:
I decided to make my html object much simpler and to allow me to do this by not using DOM and just use strings.
I am trying to create a simple alert app for some friends.
Basically i want to be able to extract data "price" and "stock availability" from a webpage like the folowing two:
http://www.sparkfun.com/commerce/product_info.php?products_id=5
http://www.sparkfun.com/commerce/product_info.php?products_id=9279
I have made the alert via e-mail and sms part but now i want to be able to get the quantity and price out of the webpages (those 2 or any other ones) so that i can compare the price and quantity available and alert us to make an order if a product is between some thresholds.
I have tried some regex (found on some tutorials, but i an way too n00b for this) but haven't managed to get this working, any good tips or examples?
$content = file_get_contents('http://www.sparkfun.com/commerce/product_info.php?products_id=9279');
preg_match('#<tr><th>(.*)</th> <td><b>price</b></td></tr>#', $content, $match);
$price = $match[1];
preg_match('#<input type="hidden" name="quantity_on_hand" value="(.*?)">#', $content, $match);
$in_stock = $match[1];
echo "Price: $price - Availability: $in_stock\n";
It's called screen scraping, in case you need to google for it.
I would suggest that you use a dom parser and xpath expressions instead. Feed the HTML through HtmlTidy first, to ensure that it's valid markup.
For example:
$html = file_get_contents("http://www.example.com");
$html = tidy_repair_string($html);
$doc = new DomDocument();
$doc->loadHtml($html);
$xpath = new DomXPath($doc);
// Now query the document:
foreach ($xpath->query('//table[#class="pricing"]/th') as $node) {
echo $node, "\n";
}
What ever you do: Don't use regular expressions to parse HTML or bad things will happen. Use a parser instead.
1st, asking this question goes too into details. 2nd, extracting data from a website might not be legitimate. However, I have hints:
Use Firebug or Chrome/Safari Inspector to explore the HTML content and pattern of interesting information
Test your RegEx to see if the match. You may need do it many times (multi-pass parsing/extraction)
Write a client via cURL or even much simpler, use file_get_contents (NOTE that some hosting disable loading URLs with file_get_contents)
For me, I'd better use Tidy to convert to valid XHTML and then use XPath to extract data, instead of RegEx. Why? Because XHTML is not regular and XPath is very flexible. You can learn XSLT to transform.
Good luck!
You are probably best off loading the HTML code into a DOM parser like this one and searching for the "pricing" table. However, any kind of scraping you do can break whenever they change their page layout, and is probably illegal without their consent.
The best way, though, would be to talk to the people who run the site, and see whether they have alternative, more reliable forms of data delivery (Web services, RSS, or database exports come to mind).
The simplest method to extract data from Website. I've analysed that my all data is covered within <h3> tag only, so I've prepared this one.
<?php
include(‘simple_html_dom.php’);
// Create DOM from URL, paste your destined web url in $page
$page = ‘http://facebook4free.com/category/facebookstatus/amazing-facebook-status/’;
$html = new simple_html_dom();
//Within $html your webpage will be loaded for further operation
$html->load_file($page);
// Find all links
$links = array();
//Within find() function, I have written h3 so it will simply fetch the content from <h3> tag only. Change as per your requirement.
foreach($html->find(‘h3′) as $element)
{
$links[] = $element;
}
reset($links);
//$out will be having each of HTML element content you searching for, within that web page
foreach ($links as $out)
{
echo $out;
}
?>