String to Zipped Stream in php - php

I have a processing server with my database and a serving database to serve up files with a low bandwidth cost. On the processing server, php is not able to create files so everything must be done with streams and/or stay in memory before being sent over to another server for download. A few days ago I found out about the stream abstraction with 'php://memory' and that I can do something like
$fp=fopen('php://memory','w+');
fwrite($fp,"Hello world");
fseek($fp,0,SEEK_SET);
//make a ftp connection here with $conn_id
$upload = ftp_fput($conn_id,"targetpath/helloworld.txt",$fp,FTP_BINARY);
to make the file in memory and then allow me to ftp it over to my other server. This is exactly what I want, except I also want to zip the data before sending it -- preferably using only native parts of php like ziparchive and not additional custom classes for special stream manipulation. I know that I am very close with the following...
$zip = new ZipArchive();
if($zip->open('php://memory', ZIPARCHIVE::CREATE)) {
$zip->addFromString('testtext.txt','Hello World!');
$fp = $zip->getStream('test'); if(!$fp) print "no filepointer";
//make a ftp connection here with $conn_id
$upload = ftp_fput($conn_id,"targetpath/helloworld.zip",$fp,FTP_BINARY);
} else print "couldn't open a zip like that";
The point at which this fails is the call to getStream (which always returns false although I think I am using correctly). It appears that the zip is fine making the file in 'php://memory' but for some reason getStream still fails although perhaps I don't sufficiently understand how ZipArchive makes zips...
How can I go from the string to the zipped filepointer so that I can ftp the zip over to my other server? Remember I can't make any files or else I would just make the zip file then ftp it over.
EDIT: based on skinnynerd's suggestions below I tried the following
$zip = new ZipArchive();
if($zip->open('php://memory', ZIPARCHIVE::CREATE)) {
$zip->addFromString('testtext.txt','Hello World!');
$zip->close();
$fp = fopen('php://memory','r+');
fseek($fp,0,SEEK_SET);
//connect to ftp
$upload = ftp_fput($conn_id,"upload/transfer/helloworld.zip",$fp,FTP_BINARY);
}
This does make a zip and send it over but the zip is 0 bytes large so I don't think that 'php://memory' works the way I thought... it actually fails at the close step -- the $zip->close() returns false which makes me wonder if I can open zips into 'php://memory' at all. Does anyone know what I can try along these line to get the zip?

$zip->getStream('test') is getting a stream to extract the file 'test' from the archive. Since there's no file 'test' in the archive, this fails. This is not the function you want to use.
As you said, what you want to do is send the finished archive to the ftp server. In this case, you would want to close the zip archive, and then reopen php://memory as a normal file (using fopen) to send it.
I don't know, but you may also be able to use $zip as a resource directly, without having to close and reopen the file.

And I think you can try create a stream pipe directly from ftp server
<?php
$zip = new ZipArchive();
if($zip->open('ftp://user:password#ftp.host.com/upload/transfer/helloworld.zip', ZipArchive::CREATE))
{
$zip->addFromString('testtext.txt','Hello World!');
$zip->close();
}
else
print "couldn't open zip file on remote ftp host.";

Does it have to be a Zip archive? Since you're trying to save bandwith it could be a gzip too.
<?php
$ftp_credentials = "ftp://USER:PASSWORD#HOST/helloworld.gz";
$gz = gzencode("Hello World!", 9);
$options = array('ftp' => array('overwrite' => true));
$stream_context = stream_context_create($options);
file_put_contents($ftp_credentials, $gz, 0, $stream_context);
?>

Related

MAMP strange behaviour : php read external file from an http:// is very slow, but from https:// is quick

I have a simple PHP script to read a remote file line-by-line, and then JSON decode it. On the production server all works ok, but on my local machine (MAMP stack, OSX) the PHP hangs. It is very slow, and takes more than 2 minutes to produce the JSON file. I think it's the json_decode() that is freezing. Why only on MAMP?
I think it's stuck in while loop, because I can't show the final $str variable that is the result of all the lines.
In case you are wondering why I need to read the file line-by-line, it's because in the real scenario, the remote JSON file is a 40MB text file. My only good performance result is like this, but any good suggestion?
Is there a configuration in php.ini to help solve this?
// The path to the JSON File
$fileName = 'http://www.xxxx.xxx/response-single.json';
//Open the file in "reading only" mode.
$fileHandle = fopen($fileName, "r");
//If we failed to get a file handle, throw an Exception.
if($fileHandle === false){
error_log("erro handle");
throw new Exception('Could not get file handle for: ' . $fileName);
}
//While we haven't reach the end of the file.
$str = "";
while(!feof($fileHandle)) {
//Read the current line in.
$line = fgets($fileHandle);
$str .= $line;
}
//Finally, close the file handle.
fclose($fileHandle);
$json = json_decode($str, true); // decode the JSON into an associative array
Thanks for your time.
I found the cause. It is path protocol.
With
$filename = 'http://www.yyy/response.json';
It freezes the server for 1 to 2 minutes.
I changed the file to another server with https protocol, and used
$filename = 'https://www.yyy/response.json';
and it works.

PHP handles a zip file as if it's empty

Here's a very stripped down version of the code I'm using.
$url = "http://server.com/getDaFile";
//Get the file from the server
$zip_file_contents = file_get_contents($url);
//Write file to disk
file_put_contents("file.zip", $zip_file_contents);
//open zip file
$zip = zip_open("file.zip");
if(is_resource($zip))
{
while($zip_entry = zip_read($zip))
{
if(zip_entry_open($zip, $zip_entry, 'r'))
{
//Read the whole file
$buf = zip_entry_read($zip_entry, zip_entry_filesize($zip_entry));
/*
Do stuff with $buf!!!
*/
zip_entry_close($zip_entry);
}
}
zip_close($zip);
}
else
{
echo "Not a resource. Oh noes!\n";
}
So : get the file, save it to disk, unzip it to extract files it contains, do stuff with files. The problem here is that, for some reason I cannot figure out, zip_read returns FALSE, as if it couldn't read files inside the ZIP archive. $zip does contain a resource, I've checked with var_dump.
What makes this even stranger is that I downloaded the ZIP file on my PC using the URL on top, manually uploaded it to the PHP server, and commented out the calls to file_get_contents and file_put_contents so PHP uses the local version. When I do this, zip_read correctly finds the right amount of files inside the ZIP and processing proceeds as it should.
I also tried doing this : $zip = zip_open($url) but $zip fails the is_resource($zip) check.
Something is obviously wrong with my code since the URL works and returns a valid ZIP archive. What is it?
So I finally found out the problem. Following #diolemo's suggestion, I opened my server's ZIP archive in a hex editor. Here's what I found at the top, followed by the usual ZIP binary data : http://pastebin.com/vQEXJtTN
It turns out there was a PHP error mixed in with the actual content of the ZIP file. Unsure of how to fix this (but knowing it certainly had to do with HTTP headers), I tried this guy's code and, what do you know, my code works perfectly now!
Lessons learned? Never trust your data, even if it seems all right (both 7-Zip and Winrar managed to open the file without problem).

Extract a file from a ZIP string

I have a BASE64 string of a zip file that contains one single XML file.
Any ideas on how I could get the contents of the XML file without having to deal with files on the disk?
I would like very much to keep the whole process in the memory as the XML only has 1-5k.
It would be annoying to have to write the zip, extract the XML and then load it up and delete everything.
I had a similar problem, I ended up doing it manually.
https://www.pkware.com/documents/casestudies/APPNOTE.TXT
This extracts a single file (just the first one), no error/crc checks, assumes deflate was used.
// zip in a string
$data = file_get_contents('test.zip');
// magic
$head = unpack("Vsig/vver/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr($data,0,30));
$filename = substr($data,30,$head['namelen']);
$raw = gzinflate(substr($data,30+$head['namelen']+$head['exlen'],$head['csize']));
// first file uncompressed and ready to use
file_put_contents($filename,$raw);
After some hours of research I think it's surprisingly not possible do handle a zip without a temporary file:
The first try with php://memory will not work, beacuse it's a stream that cannot be read by functions like file_get_contents() or ZipArchive::open(). In the comments is a link to the php-bugtracker for the lack of documentation of this problem.
There is a stream support ZipArchive with ::getStream() but as stated in the manual, it only supports reading operation on an opened file. So you cannot build a archive on-the-fly with that.
The zip:// wrapper is also read-only: Create ZIP file with fopen() wrapper
I also did some attempts with the other php wrappers/protocolls like
file_get_contents("zip://data://text/plain;base64,{$base64_string}#test.txt")
$zip->open("php://filter/read=convert.base64-decode/resource={$base64_string}")
$zip->open("php://filter/read=/resource=php://memory")
but for me they don't work at all, even if there are examples like that in the manual. So you have to swallow the pill and create a temporary file.
Original Answer:
This is just the way of temporary storing. I hope you manage the zip handling and parsing of xml on your own.
Use the php php://memory (doc) wrapper. Be aware, that this is only usefull for small files, because its stored in the memory - obviously. Otherwise use php://temp instead.
<?php
// the decoded content of your zip file
$text = 'base64 _decoded_ zip content';
// this will empty the memory and appen your zip content
$written = file_put_contents('php://memory', $text);
// bytes written to memory
var_dump($written);
// new instance of the ZipArchive
$zip = new ZipArchive;
// success of the archive reading
var_dump(true === $zip->open('php://memory'));
toster-cx had it right,you should award him the points, this is an example where the zip comes from a soap response as a byte array (binary), the content is an XML file:
$objResponse = $objClient->__soapCall("sendBill",array(parameters));
$fileData=unzipByteArray($objResponse->applicationResponse);
header("Content-type: text/xml");
echo $fileData;
function unzipByteArray($data){
/*this firts is a directory*/
$head = unpack("Vsig/vver/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr($data,0,30));
$filename = substr($data,30,$head['namelen']);
$if=30+$head['namelen']+$head['exlen']+$head['csize'];
/*this second is the actua file*/
$head = unpack("Vsig/vver/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr($data,$if,30));
$raw = gzinflate(substr($data,$if+$head['namelen']+$head['exlen']+30,$head['csize']));
/*you can create a loop and continue decompressing more files if the were*/
return $raw;
}
If you know the file name inside the .zip, just do this:
<?php
$xml = file_get_contents('zip://./your-zip.zip#your-file.xml');
If you have a plain string, just do this:
<?php
$xml = file_get_contents('compress.zlib://data://text/plain;base64,'.$base64_encoded_string);
[edit] Documentation is there: http://www.php.net/manual/en/wrappers.php
From the comments: if you don't have a base64 encoded string, you need to urlencode() it before using the data:// wrapper.
<?php
$xml = file_get_contents('compress.zlib://data://text/plain,'.urlencode($text));
[edit 2] Even if you already found a solution with a file, there's a solution (to test) I didn't see in your answer:
<?php
$zip = new ZipArchive;
$zip->open('data::text/plain,'.urlencode($base64_decoded_string));
$zip2 = new ZipArchive;
$zip2->open('data::text/plain;base64,'.urlencode($base64_string));
If you are running on Linux and have administration of the system. You could mount a small ramdisk using tmpfs, the standard file_get / put and ZipArchive functions will then work, except it does not write to disk, it writes to memory.
To have it permanently ready, the fstab is something like:
/media/ramdisk tmpfs nodev,nosuid,noexec,nodiratime,size=2M 0 0
Set your size and location accordingly so it suits you.
Using php to mount a ramdisk and remove it after using it (if it even has the privileges) is probably less efficient than just writing to disk, unless you have a massive number of files to process in one go.
Although this is not a pure php solution, nor is it portable.
You will still need to remove the "files" after use, or have the OS clean up old files.
They will of coarse not persist over reboots or remounts of the ramdisk.
if you want to read the content of a file from zip like and xml inside you shoud look at this i use it to count words from docx (wich is a zip )
if (!function_exists('docx_word_count')) {
function docx_word_count($filename)
{
$zip = new ZipArchive();
if ($zip->open($filename) === true) {
if (($index = $zip->locateName('docProps/app.xml')) !== false) {
$data = $zip->getFromIndex($index);
$zip->close();
$xml = new SimpleXMLElement($data);
return $xml->Words;
}
$zip->close();
}
return 0;
}
}
The idea comes from toster-cx is pretty useful to approach malformed zip files too!
I had one with missing data in the header, so I had to extract the central directory file header by using his method:
$CDFHoffset = strpos( $zipFile, "\x50\x4b\x01\x02" );
$CDFH = unpack( "Vsig/vverby/vverex/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr( $zipFile, $CDFHoffset, 46 ) );

PHP - Merging files with FTP

So, I'm trying to copy a file from my local server to a remote server with FTP. The problem is, I need to do this in chunks.
Thus far, based on my research, it looks like it will probably be easiest to do this by making two files on my local server: the file to be copied, and a small, temporary file that holds the current chunk. Then, I should simply merge that chunk with the remote file.
The problem is, I'm having trouble appending files. I can't figure out how the ftp:// protocol works, and I can't find a comprehensive explanation on how to do it with cURL. The closest I've found is this, but I couldn't get it working.
Below is what I've written up so far. I've commented it so you can just skim through it to get the idea, and see where I'm stuck--the code isn't complicated. What do you recommend I do? How do I append a file on my local server a file on a remote server with FTP?
<?php
// FTP credentials
$server = HIDDEN;
$username = HIDDEN;
$password = HIDDEN;
// Connect to FTP
$connection = ftp_connect($server) or die("Failed to connect to <b>$server</b>.");
// Login to FTP
if (!#ftp_login($connection, $username, $password))
{
echo "Failed to login to <b>$server</b> as <b>$username</b>.";
}
// Destination file (where the copied file should go)
$destination = 'final.txt';
// The file on my server that we're copying (in chunks) to $destination.
$read = 'readme.txt';
// Current chunk of $read.
$temp = 'temp.tmp';
// If the file we're trying to copy exists...
if (file_exists($read))
{
// Set a chunk size (this is tiny, but I'm testing
// with tiny files just to make sure it works)
$chunk_size = 4;
// For reading through the file we want to copy to the FTP server.
$read_handle = fopen($read, 'r');
// For writing the chunk to its own file.
$temp_handle = fopen($temp, 'w+');
// Loop through $read until we reach the end of the file.
while (!feof($read_handle))
{
// Read a chunk of the file we're copying.
$chunk = fread($read_handle, $chunk_size);
// Write that chunk to its own file.
fwrite($temp_handle, $chunk);
////////////////////////////////////////////
//// ////
//// NOW WHAT?? HOW DO I ////
//// WRITE / APPEND THAT ////
//// CHUNK TO $destination? ////
//// ////
////////////////////////////////////////////
}
}
fclose($read_handle);
fclose($temp_handle);
ftp_close($ftp_connect);
?>
First off i don't think you need to create the tmp file. You can append to the destination file either using "append mode" (a) as defined in the manual or use file_put_contents with the FILE_APPEND flag.
file_put_contents does takes a filename as parameter not a file handle so it basically does fopen for you which means that if you write frequently it will fopen frequently which is less optimal compared to using the file handle ad fwrite.
<?php
// The URI of the remote file to be written to
$write = 'ftp://username1:password1#domain1.com/path/to/writeme.txt';
// The URI of the remote file to be read
$read = 'ftp://username2:password2#domain2.com/path/to/readme.txt';
if (file_exists($read)) // this will work over ftp too
{
$chunk_size = 4;
$read_handle = fopen($read, 'r');
$write_handle = fopen($write, 'a');
while (!feof($read_handle))
{
$chunk = fread($read_handle, $chunk_size);
fwrite($write_handle, $chunk);
}
}
fclose($read_handle);
fclose($write_handle);
?>
As a side note: PHP has stream wrappers that make ftp access a breeze without using the ftp functions themselves.

using fgetcsv from a file on a FTP server

I need to read a list of CSV files from an FTP and delete them after I successfully read them.
Until now, i opened the csv file using fopen into a resource and then used fgetcsv to read the csv lines from it.
$res = fopen($url);
while ($csv_row = fgetcsv($res, null, self::DELIMITER)) {
.....
}
The problem is that I need to read a list of csv files and delete them too. the ftp_get function save the file into a local file. I rather avoid that. any way I can keep using the fgetcsv function with the ftp_nlist & ftp_connect functions? ?
You can save the csv file to a temporary file stream using ftp_fget(). This allows you to avoid the "create-read-delete" cycle. Once you close the file stream it's like it magically never existed :)
$ftp_handle = ftp_connect($ftp_server);
$remote_path = "/path/to/file.csv";
$tmp_handle = fopen('php://temp', 'r+');
if (ftp_fget($ftp_handle, $tmp_handle, $remote_path, FTP_ASCII)) {
rewind($tmp_handle);
while ($csv_row = fgetcsv($tmp_handle)) {
// do stuff
}
}
fclose($tmp_handle);
If you wanted to loop over a directory of files just get the list of files and then put the above code in a loop.

Categories