Server A using xmlreader to read XML from xmlwriter on server B - php

I have two servers
Server A reads http://www.some-url.com/xmlwriter_src.php using
$reader = new XMLReader();
$reader->open('http://www.some-url.com/xmlwriter_src.php');
while ($reader->read())
{
/* -- do something -- */
}
Server B creates an xml stream
$writer = new XMLWriter();
$writer->openURI('php://output');
$writer->startDocument("1.0");
$writer->startElement("records");
while(!$recordset->EOF)
{
$writer->startElement($fieldname)
$writer->text($recordset->fields[$fieldname]);
$writer->endElement();
$recordset->movenext();
}
the xmlreader on server A keeps complaining that server B doesnt respond, even though I can see the xml result in the browser.
It takes less than a second to generate
If i copy the xml to a static file, the xmlreader outputs the file.

did u tried adding
header("Content-Type: text/xml");
Else the reader will consider it as simple text and wont work. Try giving that at the begining of the file.

By default the writter will buffer your output.
Once you are done you MUST call flush().
$writer = new XMLWriter();
$writer->openURI('php://output');
$writer->startDocument("1.0");
$writer->startElement("records");
while(!$recordset->EOF)
{
$writer->startElement($fieldname)
$writer->text($recordset->fields[$fieldname]);
$writer->endElement();
$recordset->movenext();
}
$writer->flush();
By the way:
where do you close the records element?

Try writing whatever xmlReader reads on the disk and inspect the generated file. I have a hunch its either empty or invalid(incomplete) XML.
If i'm right, then you might have a timeout that expires sooner than the one you get in a real browser. Either that, or a connection that requires either connection-close or keepalive(i've seen servers broken like this).
Also, make sure you dont have a firewall on the server where the client runs that might block the xmlReader from talking to the xmlWriter:) Try iptables -L in the server console to check any firewall rules.
Edit: you might also need to call something like xmlReader->close(), or end() or whatever member you got there that closes the connection and signals the client that the transmission is over.

Related

Broadcast stream with PHP within localhost

Maybe I'm asking the impossible but I wanted to clone a stream multiple times. A sort of multicast emulation. The idea is to write every 0.002 seconds a 1300 bytes big buffer into a .sock file (instead of IP:port to avoid overheading) and then to read from other scripts the same .sock file multiple times.
Doing it through a regular file is not doable. It works only within the same script that generates the buffer file and then echos it. The other scripts will misread it badly.
This works perfectly with the script that generates the chunks:
$handle = #fopen($url, 'rb');
$buffer = 1300;
while (1) {
$chunck = fread($handle, $buffer);
$handle2 = fopen('/var/tmp/stream_chunck.tmp', 'w');
fwrite($handle2, $chunck);
fclose($handle2);
readfile('/var/tmp/stream_chunck.tmp');
}
BUT the output of another script that reads the chunks:
while (1) {
readfile('/var/tmp/stream_chunck.tmp');
}
is messy. I don't know how to synchronize the reading process of chunks and I thought that sockets could make a miracle.
It works only within the same script that generates the buffer file and then echos it. The other scripts will misread it badly
Using a single file without any sort of flow control shouldn't be a problem - tail -F does just that. The disadvantage is that the data will just accululate indefinitely on the filesystem as long as a single client has an open file handle (even if you truncate the file).
But if you're writing chunks, then write each chunk to a different file (using an atomic write mechanism) then everyone can read it by polling for available files....
do {
while (!file_exists("$dir/$prefix.$current_chunk")) {
clearstatcache();
usleep(1000);
}
process(file_get_contents("$dir/$prefix.$current_chunk"));
$current_chunk++;
} while (!$finished);
Equally, you could this with a database - which should have slightly lower overhead for the polling, and simplifies the garbage collection of old chunks.
But this is all about how to make your solution workable - it doesn't really address the problem you are trying to solve. If we knew what you were trying to achieve then we might be able to advise on a more appropriate solution - e.g. if it's a chat application, video broadcast, something else....
I suspect a more appropriate solution would be to use mutli-processing, single memory model server - and when we're talking about PHP (which doesn't really do threading very well) that means an event based/asynchronous server. There's a bit more involved than simply calling socket_select() but there are some good scripts available which do most of the complicated stuff for you.

Decompressing an LZO stream in PHP

I have a number of LZO-compressed log files on Amazon S3, which I want to read from PHP. The AWS SDK provides a nice StreamWrapper for reading these files efficiently, but since the files are compressed, I need to decompress the content before I can process it.
I have installed the PHP-LZO extension which allows me to do lzo_decompress($data), but since I'm dealing with a stream rather than the full file contents, I assume I'll need to consume the string one LZO compressed block at a time. In other words, I want to do something like:
$s3 = S3Client::factory( $myAwsCredentials );
$s3->registerStreamWrapper();
$stream = fopen("s3://my_bucket/my_logfile", 'r');
$compressed_data = '';
while (!feof($stream)) {
$compressed_data .= fread($stream, 1024);
// TODO: determine if we have a full LZO block yet
if (contains_full_lzo_block($compressed_data)) {
// TODO: extract the LZO block
$lzo_block = get_lzo_block($compressed_data);
$input = lzo_decompress( $lzo_block );
// ...... and do stuff to the decompressed input
}
}
fclose($stream);
The two TODOs are where I'm unsure what to do:
Inspecting the data stream to dtermine whether I have a full LZO block yet
Extracting this block for decompression
Since the compression was done by Amazon (s3distCp) I don't have control over the block size, so I'll probably need to inspect the incoming stream to determine how big the blocks are -- is this a correct assumption?
(ideally, I'd use a custom StreamFilter directly on the stream, but I haven't been able to find anyone who has done that before)
Ok executing a command via PHP can be done in many different ways, something like:
$command = 'gunzip -c /path/src /path/dest';
$escapedCommand = escapeshellcmd($command);
system($escapedCommand);
or also
shell_exec('gunzip -c /path/src /path/dest');
will do the work.
Now it's a matter of what command to execute, under Linux there's a nice command line tool called lzop which extracts orcompresses lzop files.
You can use it via something like:
lzop -dN sources.lzo
So you final code might be something as easy as:
shell_exec('lzop -dN s3://my_bucket/my_logfile');

Validating a large XML file ~400MB in PHP

I have a large XML file (around 400MB) that I need to ensure is well-formed before I start processing it.
First thing I tried was something similar to below, which is great as I can find out if XML is not well formed and which parts of XML are 'bad'
$doc = simplexml_load_string($xmlstr);
if (!$doc) {
$errors = libxml_get_errors();
foreach ($errors as $error) {
echo display_xml_error($error);
}
libxml_clear_errors();
}
Also tried...
$doc->load( $tempFileName, LIBXML_DTDLOAD|LIBXML_DTDVALID )
I tested this with a file of about 60MB, but anything a lot larger (~400MB) causes something which is new to me "oom killer" to kick in and terminate the script after what always seems like 30 secs.
I thought I may need to increase the memory on the script so figured out the peak usage when processing 60MB and adjusted it accordingly for a large and also turn the script time limit off just in case it was that.
set_time_limit(0);
ini_set('memory_limit', '512M');
Unfortunately this didn't work, as oom killer appears to be a linux thing that kicks in if memory load (even the right term?) is consistently high.
It would be great if I could load xml in chunks somehow as I imagine this will reduce the memory load so that oom killer doesn't stick it's fat nose in and kill my process.
Does anyone have any experience validating a large XML file and capturing errors of where it's badly formed, a lot of posts I've read point to SAX and XMLReader that might solve my problem.
UPDATE
So #chiborg pretty much solved this issue for me...the only downside to this method is that I don't get to see all of the errors in the file, just the first that failed which I guess makes sense as I think it can't parse past the first point that fails.
When using simplexml...it's able to capture most of the issues in the file and show me at the end which was nice.
Since the SimpleXML and DOM APIs will always load the document into memory, using a streaming parser like SAX or XMLReader is the better approach.
Adpating the code from the example page, it could look like this:
$xml_parser = xml_parser_create();
if (!($fp = fopen($file, "r"))) {
die("could not open XML input");
}
while ($data = fread($fp, 4096)) {
if (!xml_parse($xml_parser, $data, feof($fp))) {
$errors[] = array(
xml_error_string(xml_get_error_code($xml_parser)),
xml_get_current_line_number($xml_parser));
}
}
xml_parser_free($xml_parser);
For big file, perfect use XMLReader class.
But if liked simplexml syntax: https://github.com/dkrnl/SimpleXMLReader/blob/master/library/SimpleXMLReader.php
Usage example: http://github.com/dkrnl/SimpleXMLReader/blob/master/examples/example1.php

saving and reading a xml file getting from the other url

may be i am going to ask some stupid question but i don't have any idea about php
that's why i want to know it i never worked in php and now i have to do it so please provide me some useful tips,
i have XML file that is coming from a different URL and i want to save it on the server then i have to read it and extract it to a page in proper format and some modification in data.
You can use DOM
$dom = new DOMDocument();
$dom->load('http://www.example.com');
This would load the XML from the remote URL. You can then process it as needed. See my previous answers on various topics using DOM. To save the file to your server after your processed it, you use
$dom->save('filename.xml');
Loading the file with $dom->load() will only work if you have allow_url_fopen enabled in your php.ini. If not, you have to use cURL to download the remote file first.
Maybe this should be helpfull to you: http://www.php.net/manual/en/function.simplexml-load-file.php
If you're have dificulte to get the XML file from the remote host you can use combine with above simplexml-load-string
$path_to_xml = 'http://some.com/file.xml';
$xml = simplexml_load_string( file_get_content($path_to_xml) );

Parsing XML file through PHP while using XSLT as the main template file

I have a lots (500ish) xml files from An old ASP and VBscript that was running on an old windows server. The user could click a link to download the requested xml file, or click a link to view how the xml file will look, once its imported into their system...
If clicked to view the output, this opened a popup window were the xml filename is passed via URL & using the xslt template file this would display the output.
example url = /transform.php?action=transform&xmlProtocol=AC_Audiology.xml
Now were using PHP5 im trying to get something that resembles the same output.
we started looking into xslt_create(); but this is an old function from php4
I'm looking for the best method to deploy this.
The main php page should check & capture the $_GET['xmlProtocol'] value.
pass this to the xslt template page as data;
were it will be output in html.
a general point in the right direction would be great!
You can find the documentation (+examples) of the "new" XSL(T) extension at http://docs.php.net/xsl.
php
// Transform.php
if(isset($_GET['action']) && $_GET['action'] == 'transform') {
// obviously you would never trust the input and would validate first
$xml_file = AFunctionValidateAndGetPathToFile($_GET['xmlProtocol']);
// Load up the XML File
$xmlDoc = new DOMDocument;
$xmlDoc->load($xml_file);
// Load up the XSL file
$xslDoc = new DomDocument;
$xslDoc->load("xsl_template_file.xsl");
$xsl = new XSLTProcessor;
$xsl->importStyleSheet($xslDoc);
// apply the transformation
echo $xsl->transformToXml($xmlDoc);
}
I had a similar problem about two years ago. I was using PHP5 but needed to use xslt_create(); or an equivalent. Ultimately, I switched to PHP4.
You can probably set your server to use PHP5 everywhere except for files in a certain folder. I believe that's what I did so I could process XSL files using PHP4 but the majority of the site still used PHP5.
It's possible that things have changed in the last two years and PHP5 has better support for something like xslt_create(); ---- I haven't been following recent changes.
Hope this helps!

Categories