HTML parser to a website chart [closed] - php

Question

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 10 years ago.
So, there is this website where they have some statistics about that website, and it's updated daily. What I want to do is to scrap that website and retrieve those informations to my website and put them into a line chart. Something like these http://code.google.com/apis/ajax/playground/?type=visualization#line_chart
or these http://code.google.com/apis/ajax/playground/?type=visualization#image_line_chart
The only problem is that I don't know how to do it.

Answer 1

Get Permission
If you need to crawl a remote page, you should start by making sure you're not doing anything illegal. I'm in no way capable of counseling you about the legality of what you're looking to do, but I can say with assurance that some people don't like having their data scrapped - you really should check with the site administrators to see if they have any problem with this.
Got Permission? Good...
When you're cleared, consider using the DOMDocument class to create a representation of the DOM in a series of nested objects that you can interact with pretty trivially:
$doc = new DOMDocument();
$doc->loadHTML(file_get_contents("http://foo.com/data/statistics"));
If you're at all familiar with DOM-traversal and selection methods in JavaScript, the DOMDocument interface will seem pretty familiar. In order to get a particular element by ID, you'd use the appropriate method:
$statistics = $doc->getElementById("statistics");
And to get all TD elements within that element:
$cells = $statistics->getElementsByTagName("td");
Without going into too much detail, you could continue to use the methods provided to you by the DOMDocument class to traverse and select data. When you get to the actual nodes you're seeking, you could easily save their values into an array, and then output that array as a string with some JavaScript to show the Google graph.
Be Nice!
It would be wise to cache the results of this operation so that you don't hit the remote server every time the script runs. Save the output to a local file with a timestamp, and check that timestamp for expiration - when it's expired, remove it, and create a new cached result in its place.
What This Might Look Like
Here is a very basic implementation of what your solution may resemble when it is complete. Note that we only hit the remote server at most once per day.
// Silence errors due to malformed HTML
libxml_use_internal_errors(true);
// This function builds out data out, when necessary
function build_output () {
// Create a DOMDocument, grab debt-clock HTML
$doc = new DOMDocument();
$doc->loadHTML(file_get_contents("http://brillig.com/debt_clock/"));
// Find element representing total National Debt
$total = $doc->getElementsByTagName("img")->item(0);
// Grab value from the alt attribute
$total = $total->attributes->getNamedItem("alt")->nodeValue;
// Second paragraph has two more values of interest
$parag = $doc->getElementsByTagName("p")->item(1);
// Build out resulting array of data: Natl. Debt, Population, Ind. Debt
$data = Array(
"ntl" => str_replace(" ", "", $total),
"pop" => $parag->getElementsByTagName("b")->item(0)->nodeValue,
"pay" => $parag->getElementsByTagName("b")->item(1)->nodeValue
);
// Return a JSON string of this data
return json_encode($data);
}
// Most recent cache file (today)
$cache_name = date("Y-m-d");
if (!file_exists($cache_name)) {
// Today's cache doesn't exist, create it.
$output = build_output();
$message = "Fresh: " . $output;
file_put_contents($cache_name, $output);
} else {
// Today has already been cached, load it.
$message = "Cached: " . file_get_contents($cache_name);
}
// Show the user the output
echo $message;
When loading from cache, the output from the above script looks similar to this:
Cached: {
"ntl":"$16,293,644,693,599.87",
"pop":"313,929,808",
"pay":"$51,902.19"
}
You can load that JSON data into any service you like now to generate an image representing its data.