I am reading an XML feed and would like to compare to an old version to check for updates.
My problems at the moment are that I can't seem to make a copy of a SimpleXML object, and the other problem is I'm not sure I can directly compare them.
This is my code as it stands. I'm obviously just testing on local files, but I intend to eventually load from the web.
Is it okay to use sleep for very long periods? I was thinking 15 minute interval is often enough for my purpose.
error_reporting(E_NOTICE);
$file = 'tmbdata_sm.xml';
$xml_old = "";
while(true){
$xml = simplexml_load_file($file);
if($xml != $xml_old){
foreach($xml->channel->item as $item){
echo $item->title . "\n";
echo $item->link . "\n";
}
$xml_old = clone $xml;
$xml = "";
}else{
echo 'no change';
}
sleep(60);
}
Without a definition of what "updated" means in your context I'm afraid your question may remain unanswered. String compare might work, but a better and faster way would be to use filemtime() which lets you know the last time that the file was modified.
Also you should refrain from using sleep() in an infinite loop like you are doing. I don't think having PHP running indefinitely will be healthy for your computer or your server. The proper way to do this is either a cronjob when using UNIX, or the task scheduler when in Windows.
I think you can't compare simple xml objects in this way.
I would try to download the xml using whatever you feel comfortable with (say, cURL extension), then compare the xml text strings, and then when you find they are different, use simplexml_load_string() to parse the xml text.
Related
I'm working on some system for a few hours now and this little thing is too much for me to think logically about at the moment.
Normally I would wait a few hours but this is a last minute job and I need to finish this.
Here's my problem:
I have an XML file that gets posted to my PHP file, the PHP file inserts certain data into a DB, but some XML nodes have the same name:
<accessoires>
<accessoire>value1</accessoire>
<accessoire>value2</accessoire>
<accessoire>value3</accessoire>
</accessoires>
Now I want to get a var $acclist which contains all values seperated by a comma:
value1,value2,value3,
I bet the solution to this is very easy but I'm at the known point where even the easiest piece of code becomes a hassle. And googling only comes up with nodes that in some way have their own identifiers.
Could someone help me out please?
You can try simplexml_load_string to parse the html then call implode on the node after casting to an array.
NOTE This code was tested in php 5.4.6 and behaves as expected.
<?php
$xml = '<accessoires>
<accessoire>value1</accessoire>
<accessoire>value2</accessoire>
<accessoire>value3</accessoire>
</accessoires>';
$dat = simplexml_load_string($xml);
echo implode(",",(array)$dat->accessoire);
For 5.3.x I had to change to
$xml = '<accessoires>
<accessoire>value1</accessoire>
<accessoire>value2</accessoire>
<accessoire>value3</accessoire>
</accessoires>';
$dat = simplexml_load_string($xml);
$dat = (array)$dat;
echo implode(",",$dat["accessoire"]);
You do this by taking a library that is able to parse and process XML, for example with SimpleXML:
implode(',', iterator_to_array($accessoires->accessoire, FALSE));
The key part here is to use iterator_to_array() as SimpleXML offers the same-named child-elements here as an iterator. Otherwise $accessoires->accessoire gives you auto-magically only the first element (if any).
I want to extrat the content of a specific div in an external webpage, the div looks like this:
<dt>Win rate</dt><dd><div>50%</div></dd>
My target is the "50%". I'm actually using this php code to extract the content:
function getvalue($parameter,$content){
preg_match($parameter, $content, $match);
return $match[1];
};
$parameter = '#<dt>Score</dt><dd><div>(.*)</div></dd>#';
$content = file_get_contents('https://somewebpage.com');
Everything works fine, the problem is that this method is taking too much time, especially if I've to use it several times with diferents $content.
I would like to know if there's a better (faster, simplier, etc.) way to acomplish the same function? Thx!
You may use DOMDocument::loadHTML and navigate your way to the given node.
$content = file_get_contents('https://somewebpage.com');
$doc = new DOMDocument();
$doc->loadHTML($content);
Now to get to the desired node, you may use method DOMDocument::getElementsByTagName, e.g.
$dds = $doc->getElementsByTagName('dd');
foreach($dds as $dd) {
// process each <dd> element here, extract inner div and its inner html...
}
Edit: I see a point #pebbl has made about DomDocument being slower. Indeed it is, however, parsing HTML with preg_match is a call for trouble; In that case, I'd also recommend looking at event-driven SAX XML parser. It is much more lightweight, faster and less memory intensive as it does not build a tree. You may take a look at XML_HTMLSax for such a parser.
There are basically three main things you can do to improve the speed of your code:
Off load the external page load to another time (i.e. use cron)
On a linux based server I would know what to suggest but seeing as you use Windows I'm not sure what the equivalent would be, but Cron for linux allows you to fire off scripts at certain schedule time offsets - in the background - so not using a browser. Basically I would recommend that you create a script who's sole purpose is to go and fetch the website pages at a particular time offset (depending on how frequently you need to update your data) and then write those webpages to files on your local system.
$listOfSites = array(
'http://www.something.com/page.htm',
'http://www.something-else.co.uk/index.php',
);
$dirToContainSites = getcwd() . '/sites';
foreach ( $listOfSites as $site ) {
$content = file_get_contents( $site );
/// i've just simply converted the URL into a filename here, there are
/// better ways of handling this, but this at least keeps things simple.
/// the following just converts any non letter or non number into an
/// underscore... so, http___www_something_com_page_htm
$file_name = preg_replace('/[^a-z0-9]/i','_', $site);
file_put_contents( $dirToContainSites . '/' . $file_name, $content );
}
Once you've created this script, you then need to set the server up to execute it as regularly as you need. Then you can modify your front-end script that displays the stats to read from local files, this would give a significant speed increase.
You can find out how to read files from a directory here:
http://uk.php.net/manual/en/function.dir.php
Or the simpler method (but prone to possible problems) is just to re-step your array of sites, convert the URLs to file names using the preg_replace above, and then check for the file's existence in the folder.
Cache the result of calculating your statistics
It's quite likely this being a stats page that you'll want to visit it quite frequently (not as frequent as a public page, but still). If the same page is visited more often than the cron-based script is executed then there is no reason to do all the calculation again. So basically all you have to do to cache your output is do something similar to the following:
$cachedVersion = getcwd() . '/cached/stats.html';
/// check to see if there is a cached version of this page
if ( file_exists($cachedVersion) ) {
/// if so, load it and echo it to the browser
echo file_get_contents($cachedVersion);
}
else {
/// start output buffering so we can catch what we send to the browser
ob_start();
/// DO YOUR STATS CALCULATION HERE AND ECHO IT TO THE BROWSER LIKE NORMAL
/// end output buffering and grab the contents so we now have a string
/// of the page we've just generated
$content = ob_get_contents(); ob_end_clean();
/// write the content to the cached file for next time
file_put_contents($cachedVersion, $content);
echo $content;
}
Once you start caching things you need to be aware of when you should delete or clear your cache - otherwise if you don't your stats output will never change. With regards to this situation, the best time to clear your cache is at the point you go and fetch the external web pages again. So you should add this line to the bottom of your "cron" script.
$cachedVersion = getcwd() . '/cached/stats.html';
unlink( $cachedVersion ); /// will delete the file
There are other speed improvements you could make to the caching system (you could even record the modified times of the external webpages and load only when they have been updated) but I've tried to keep things easy to explain.
Don't use a HTML Parser for this situation
Scanning a HTML file for one particular unique value does not require the use of a fully-blown or even lightweight HTML Parser. Using RegExp incorrectly seems to be one of those things that lots of start-up programmers fall into, and is a question that is always asked. This has led to lots of automatic knee-jerk reactions from more experience coders to automatically adhere to the following logic:
if ( $askedAboutUsingRegExpForHTML ) {
$automatically->orderTheSillyPersonToUse( $HTMLParser );
} else {
$soundAdvice = $think->about( $theSituation );
print $soundAdvice;
}
HTMLParsers should be used when the target within the markup is not so unique, or your pattern to match relies on such flimsy rules that it'll break the second an extra tag or character occurs. They should be used to make your code more reliable, not if you want to speed things up. Even parsers that do not build a tree of all the elements will still be using some form of string searching or regular expression notation, so unless the library-code you are using has been compiled in an extremely optimised manner, it will not beat well coded strpos/preg_match logic.
Considering I have not seen the HTML you are hoping to parse, I could be way off, but from what I've seen of your snippet it should be quite easy to find the value using a combination of strpos and preg_match. Obviously if your HTML is more complex and might have random multiple occurances of <dt>Win rate</dt><dd><div>50%</div></dd> it will cause problems - but even so - a HTMLParser would still have the same problem.
$offset = 0;
/// loop through the occurances of 'Win rate'
while ( ($p = stripos ($html, 'win rate', $offset)) !== FALSE ) {
/// grab out a snippet of the surrounding HTML to speed up the RegExp
$snippet = substr($html, $p, $p + 50 );
/// I've extended your RegExp to try and account for 'white space' that could
/// occur around the elements. The following wont take in to account any random
/// attributes that may appear, so if you find some pages aren't working - echo
/// out the $snippet var using something like "echo '<xmp>'.$snippet.'</xmp>';"
/// and that should show you what is appearing that is breaking the RegExp.
if ( preg_match('#^win\s+rate\s*</dt>\s*<dd>\s*<div>\s*([0-9]+%)\s*<#i', $snippet, $regs) ) {
/// once you are here your % value will be in $regs[1];
break; /// exit the while loop as we have found our 'Win rate'
}
/// reset our offset for the next loop
$offset = $p;
}
Gotchas to be aware of
If you are new to PHP, as you state in a comment above, then the above may seem rather complicated - which it is. What you are trying to do is quite complex, especially if you want to do it optimally and fast. However, if you follow throught the code I've given and research any bits that you aren't sure of / haven't heard of (php.net is your friend), it should give you a better understanding of a good way to achieve what you are doing.
Guessing ahead however, here are some of the problems you might face with the above:
File Permission errors - in order to be able to read and write files to and from the local operating system you will need to have the correct permissions to do so. If you find you can not write files to a particular directory it might be that the host you are using wont allow you to do so. If this is the case you can either contact them to ask about how to get write permission to a folder, or if that isn't possible you can easily change the code above to use a database instead.
I can't see my content - when using output buffering all the echo and print commands do not get sent to the browser, they instead get saved up in memory. PHP should automatically output all the stored content when the script exits, but if you use a command like ob_end_clean() this actually wipes the 'buffer' so all the content is erased. This can lead to confusing situations when you know you are echoing something.. but it just isn't appearing.
(Mini Disclaimer :) I've typed all the above manually so you may find there are PHP errors, if so, and they are baffling, just write them back here and StackOverflow can help you out)
Instead of trying to not use preg_match why not just trim your document contents down in size? for example, you could dump everything before <body and everything after </body>. then preg_match will be searching less content already.
Also, you could try to do each one of these processes as a pseudo separate thread, so that way they aren't happening one at a time.
I have this problem in terms of processing time of a large xml files. By large, i mean 600MB on the average.
Currently, It takes about 50 - 60 minutes to parse and insert the data into a database.
I would like to ask for suggestions on how can I improve the processing time? Like goind down to 20 minutes.
Because with the current time it will take me 2.5 months to populate the database with the content from the xml. By the way I have 3000+ xml files with the average of 600mb. And my php script in command line thru cron job.
I have also read other questions like the one below, but I have not found any idea yet.
What is the fastest XML parser in PHP?
I see that some have parsed files up to 2GB. I wonder how long are the processing time.
I hope you guys could lend your help.
It would be much appreciated.
Thanks.
I have this code:
$handler = $this;
$parser = xml_parser_create('UTF-8');
xml_set_object($parser, $handler);
xml_parser_set_option($parser, XML_OPTION_CASE_FOLDING, false);
xml_set_element_handler($parser, "startElement", "endElement");
xml_set_character_data_handler($parser, "cdata");
$fp = fopen($xmlfile, 'r');
while (!feof($fp)) {
while (($data = fread($fp, 71680))){
}
}
I put first the parse data in a temporary array.
My mysql insert commands are inside the endElement function.
There is a specific closing tag to trigger my insert command to the database.
Thanks for the response....
Without seeing any code, the very first thing I have to suggest is NOT to use either DOM or SimpleXMLElement as these load the whole thing into memory.
You need to use a stream parser like XMLReader.
EDIT:
Since you are already using a stream parser, you aren't going to get huge gains from changing parsers (I honestly don't know the difference in speed between XML Parser and XMLReader, since the latter uses libxml, it may be better but probably not worth it).
Next thing to look at is whether you're doing anything silly in your code; for that we'd need to see a more substantial overview of how you've implemented this.
You say you are putting data in a temporary array and calling MySQL insert once you reach a closing tag. Are you using prepared statements? Are you using transactions to do multiple inserts in bulk?
The right way to get at your bottleneck though is to run a profiler over your code. My favourite tool for the job is xhProf with XHGui. This will tell you what functions are running, how many times, for how long and how much memory they consume (and can display it all in a nice call-graph, very useful).
Use the instructions on that GitHub's README. Here's a tutorial and another useful tutorial (bear in mind this last one is for the profiler without the XHGui extensions that I linked to).
You only seem to need to parse and read the data and not edit the XML. With this mind, I would say using a SAX parser is the easier and faster way to do this.
SAX is an approach to parse XML documents, but not to validate them. The good thing is that you can use it with both PHP 4 and PHP 5 with no changes. In PHP 4, the SAX parsing is already available on all platforms, so no separate installation is necessary.
You basically define a function to be run when a start element is found and another to be run when an end element is found (you can also use one for attributes). And then you do whatever you want with the parsed data.
Parsing XML with SAX
<?
function start_element($parser, $element_name, $element_attrs) {
switch ($element_name) {
case 'KEYWORDS':
echo '<h1>Keywords</h1><ul>';
break;
case 'KEYWORD':
echo '<li>';
break;
}
}
function end_element($parser, $element_name) {
switch ($element_name) {
case 'KEYWORDS':
echo '</ul>';
break;
case 'KEYWORD':
echo '</li>';
break;
}
}
function character_data($parser, $data) {
echo htmlentities($data);
}
$parser = xml_parser_create();
xml_set_element_handler($parser, 'start_element', 'end_element');
xml_set_character_data_handler($parser, 'character_data');
$fp = fopen('keyword-data.xml', 'r')
or die ("Cannot open keyword-data.xml!");
while ($data = fread($fp, 4096)) {
xml_parse($parser, $data, feof($fp)) or
die(sprintf('XML ERROR: %s at line %d',
xml_error_string(xml_get_error_code($parser)),
xml_get_current_line_number($parser)));
}
xml_parser_free($parser);
?>
Source: I worked on parsing and processing large amounts of XML data.
EDIT: Better example
EDIT: Well, apparently you are already using a Sax Parser. As long as you are actually processing the file in an event driven way (Not having any additional overhead) you should be at top performance in this department. I would say there is nothing you can do to increase the parsing performance. If you are having performance issues I would suggest looking at what you are doing in your code to find performance bottlenekcs (Try using a php profiler like this one ). If you post your code here, we could give it a look! Cheers!
I have spent the last day or so tackling the same problem. I noticed that limiting the number of insert queries reduced the processing time quite significantly. You might have already done this but try collecting a batch of parsed data into a suitable data structure (I am using a simple array, but maybe a more suitable data structure could further reduce the cost?). Upon a collection of X sets insert the data in one go (INSERT INTO table_name (field_name) VALUES (set_1, set_2, set_n) )
Hope this helps anyone who might stumble upon this page. I am still working out other bottlenecks, if I find something new I will post it here.
What is more efficient and/or what is better practice, to echo the HTML or have many open and close php tags?
Obviously for big areas of HTML it is sensible to open and close the php tags. What about when dealing with something like generating XML? Should you open and close the php tags with a single echo for each piece of data or use a single echo with the XML tags included in quotations?
From a maintenance perspective, one should have the HTML / XML as separate from the code as possible IMO, so that minor changes can be made easily even by a non-technical person.
The more a homogeneous block the markup is, the cleaner the work.
One way to achieve this is to prepare as much as possible in variables, and using the heredoc syntax:
// Preparation
$var1 = get_value("yxyz");
$var2 = get_url ("abc");
$var3 = ($count = 0 ? "Count is zero" : "Count is not zero");
$var4 = htmlentities(get_value("def"));
// Output
echo <<<EOT
<fieldset title="$var4">
<ul class="$var1">
<li>
$var2
</li>
</ul>
</fieldset>
EOT;
You will want to use more sensible variable names, of course.
Edit: The link pointed out by #stesch in the comments provides some good arguments towards using a serializer when producing XML, and by extension, even HTML, instead of printing it out as shown above. I don't think a serializer is necessary in every situation, especially from a maintenance standpoint where templates are so much more easy to edit, but the link is well worth a read. HOWTO Avoid Being Called a Bozo When Producing XML
Another big advantage of the separation between logic and content is that if transition to a templating engine, or the introduction of caching becomes necessary one day, it's almost painless to implement because logic and code are already separated.
PHP solves this problem by what is known as heredocs. Check it out please.
Example:
echo <<<EOD
<td class="itemname">{$k}s</td>
<td class="price">{$v}/kg</td>
EOD;
Note: The heredoc identifer (EOD in this example) must not have any spaces or indentation.
Whichever makes sense to you. The performance difference is marginal, even if a large echo is faster.
But an echo of a big string is hard to read and more <?php echo $this->that; ?> tell a story :)
echo sends its argument further down the request processing chain, and eventually this string is sent to the client through a say, network socket. Depending on how the echo works in conjunction with underlying software layers (e.g. webserver), sometimes your script may be able to execute faster than it can push data to the client. Without output buffering, that is. With output buffering, you trade memory to gain speed - you echos are faster because they accumulate in a memory buffer. But only if there is no implicit buffering going on. One'll have to inspect Apache source code to see how does it treat PHPs stdout data.
That said, anything below is true for output buffering enabled scripts only, since without it the more data you attempt to push at once the longer you have to wait (the client has to receive and acknowledge it, by ways of TCP!).
It is more efficient to send a large string at once than do N echos concatenating output. By similar logic, it is more efficient for the interpreter to enter the PHP code block (PHP processing instruction in SGML/XML markup) once than enter and exit it many times.
As for me, I assemble my markup not with echo, but using XML DOM API. This is also in accordance with the article linked above. (I reprint the link: http://hsivonen.iki.fi/producing-xml/) This also answers the question whether to use one or many PHP tags. Use one tag which is your entire script, let it assemble the resulting markup and send it to the client.
Personally I tend to prefer what looks the best as code readability is very important, particularly in a team environment. In terms of best practice I'm afraid I'm not certain however it is usually best practice to optimize last meaning that you should write it for readability first and then if you encounter speed issues do some refactoring.
Any issues you have with efficiency are likely to be elsewhere in your code unless you are doing millions of echo's.
Another thing to consider is the use of an MVC to separate your "views" from all of your business logic which is a very clean way to code. Using a template framework such as smarty can take this one step further leading to epic win.
Whatever you do, don't print XML!
See HOWTO Avoid Being Called a Bozo When Producing XML
I've made myself the same question long time ago and came up with the same answer, it's not a considerable difference. I deduct this answer with this test:
<?
header('content-type:text/plain');
for ($i=0; $i<10; $i++) {
$r = benchmark_functions(
array('output_embeed','output_single_quote','output_double_quote'),
10000);
var_dump($r);
}
function output_embeed($i) {
?>test <?php echo $i; ?> :)<?
}
function output_single_quote($i) {
echo 'test '.$i.' :)';
}
function output_double_quote($i) {
echo "test $i :)";
}
function benchmark_functions($functions, $amount=1000) {
if (!is_array($functions)||!$functions)
return(false);
$result = array();
foreach ($functions as $function)
if (!function_exists($function))
return(false);
ob_start();
foreach ($functions as $idx=>$function) {
$start = microtime(true);
for ($i=0;$i<$amount;$i++) {
$function($idx);
}
$time = microtime(true) - $start;
$result[$idx.'_'.$function] = $time;
}
ob_end_clean();
return($result);
}
?>
I know how to do this in Ruby, but I want to do this in PHP. Grab a page and be able to parse stuff out of it.
Take a look at cURL. Knowing about cURL and how to use it will help in many ways as it's not specific to PHP. If you want something specific however, you can use file_get_contents which is the recommended way in PHP to get the contents of a file into a string.
$file = file_get_contents("http://google.com/");
How to parse it depends on what you are trying to do, but I'd recommend one of the XML libraries for PHP.
You could use fopen in read mode: fopen($url, 'r'); or more simply file_get_contents($url);. You could also use readfile(), but file_get_contents() is potentially more efficient and therefore recommended.
Note: these are dependent on config (see the linked manual page) but will work on most setups.
For parsing, simplexml is enabled by default in PHP.
$xmlObject = simplexml_load_string($string);
// If the string was valid, you now have a fully functional xml object.
echo $xmlObject->username;
Its funny, I had the opposite question when I started rails development