I have a project where the user can add multiple files to a cart and then zip them into a single file and download all of them at once.
It seems though that as soon as the cart gets bigger then 10mb it cannot create the zip file.
I thought this might be because of upload_max_filesize, but that is set to 200m and same for post_max_size.
What would be limiting this?
Here is my code.
It creates a random file name, then checks the cart loop and adds each file.
// create object
$zip = new ZipArchive();
$filename = "loggedin/user_files/ABX-DOWNLOAD-".rand(1, 1000000).".zip";
while (file_exists($filename)):
$filename = "loggedin/user_files/ABX-DOWNLOAD-".rand(1, 1000000).".zip";
endwhile;
// open archive
if ($zip->open('../'.$filename, ZIPARCHIVE::CREATE) !== TRUE) {
die ("Could not open archive");
}
// add files
//echo $all_audio;
foreach($audio_ids as $audio_id){
if($audio_id != ""){
$audio = Audio::find_by_id($audio_id);
$selected_category = Categories::find_by_id($audio->category_id);
$selected_sub_category = Sub_Categories::find_by_id($audio->sub_category_id);
$selected_sub_sub_category = Sub_Sub_Categories::find_by_id($audio->sub_sub_category_id);
$f = "uploads/mp3/".$selected_category->category_name."/".$selected_sub_category->sub_category_name."/".$selected_sub_sub_category->sub_sub_category_name."/".$audio->media_path;
$zip->addFile($f) or die ("ERROR: Could not add file: $f");
}
}
// close and save archive
$zip->close() or die("ERROR: COULD NOT CLOSE");
}
If the archive is being created in memory, you can increase the PHP memory limit through PHP.ini or other means, or alternatively write the file directly to disk while it is being created.
You can also stream the file to the user as it is being created. See php rendering large zip file - memory limit reached
Are you using the addFile() method?
If so, try replacing that call with addFromString() + file_get_contents().
Well, narrowed it down. A pathname was corrupted and was making the zip error out because the file did not exist. Looks like I need some better error checking.
Related
I am having trouble appending to my log file. If my log file is over 50 mb, then make a new log file. If not, then append to the previous file. I am not sure if I am using FILE_APPEND correctly. Would it be best to use fopen and fwrite?
Here is my code below:
//this is the file to test size to determine whether to append to it or start a new log file
if ($latestFile != '') {
$latestFileSize = filesize($latestFile);//file size in bytes (1/1000000 of MB)
}
if ($latestFileSize != 0) {
$fileSizeThreshold = 50 * 1000000;//Threshold for log file size limit in bytes (50 MB)
if ($latestFileSize > $fileSizeThreshold) {
//The latest results file deletion log file is over 50 MB in size, so create a new one
file_put_contents("c:\\sites\\{$logFileName}", $log);
} else {
//2. Troubleshoot why the FILE_APPEND is not working and fix it. What is the normal behavior of the FILE_APPEND?
//The latest results file deletion log file is NOT over 50 MB in size, so append log entry to latest one
file_put_contents("c:\\sites\\{$latestFileName}", $log, FILE_APPEND);
}
} else {
//This is the first file in the log directory. Create it.
file_put_contents("c:\\sites\\{$logFileName}", $log);
}
closedir($handle);
}//end if $handle
Are you sure $latestFileSize is not always == 0?
Nothing wrong with file_put_contents usually, troubleshoot the logic first. Make sure you do get to the file_put_contents with FILE_APPEND, as the way you use it is perfectly fine.
If you know the size of the file function filesize($file_path)
it will have a size A bytes.
Open the file and output the contents in textarea.
Without making any changes, save it:
If you then open the file
$save_file = fopen($file_path, 'w');
if ($save_file)
{
// Write new data to a file
$do_write = fwrite ($save_file, stripslashes ($_POST['file']));
if ($do_write)
{
$message = "Success";
}
else
{
$message = "Error";
}
}
else
{
$message = "Error";
}
Look again at the file size function filesize($file_path) it will have size B bytes.
Where A < B.
Problem: Why do I get the size of the file if the file itself and its contents have not been altered?
Writing to a file with the 'w' permission will append to the file if it already has content. Also keep in mind that the length of the file name effects the number of bytes a file uses.
For example, backing up file.ext as file.ext.bak will make file.ext.bak 3 bytes longer.
You may be adding whitespace to the file and not know it. I would try and track the changes with git and see what is changing. You can also do a diff in shell. a A good way to check if a file has changed without checking the filesize is by hashing it. Here is what you do:
$hash = hash_file('md5', '/internal/path/to/file.txt');
// Code to store hash code in database/file.
Then to check if it has changed do the same thing and compare it to the stored hash. This will be more reliable than filesize since meta-data (name, extension, permissions) on the file could change filesize and leave the file contents unchanged.
I have a BASE64 string of a zip file that contains one single XML file.
Any ideas on how I could get the contents of the XML file without having to deal with files on the disk?
I would like very much to keep the whole process in the memory as the XML only has 1-5k.
It would be annoying to have to write the zip, extract the XML and then load it up and delete everything.
I had a similar problem, I ended up doing it manually.
https://www.pkware.com/documents/casestudies/APPNOTE.TXT
This extracts a single file (just the first one), no error/crc checks, assumes deflate was used.
// zip in a string
$data = file_get_contents('test.zip');
// magic
$head = unpack("Vsig/vver/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr($data,0,30));
$filename = substr($data,30,$head['namelen']);
$raw = gzinflate(substr($data,30+$head['namelen']+$head['exlen'],$head['csize']));
// first file uncompressed and ready to use
file_put_contents($filename,$raw);
After some hours of research I think it's surprisingly not possible do handle a zip without a temporary file:
The first try with php://memory will not work, beacuse it's a stream that cannot be read by functions like file_get_contents() or ZipArchive::open(). In the comments is a link to the php-bugtracker for the lack of documentation of this problem.
There is a stream support ZipArchive with ::getStream() but as stated in the manual, it only supports reading operation on an opened file. So you cannot build a archive on-the-fly with that.
The zip:// wrapper is also read-only: Create ZIP file with fopen() wrapper
I also did some attempts with the other php wrappers/protocolls like
file_get_contents("zip://data://text/plain;base64,{$base64_string}#test.txt")
$zip->open("php://filter/read=convert.base64-decode/resource={$base64_string}")
$zip->open("php://filter/read=/resource=php://memory")
but for me they don't work at all, even if there are examples like that in the manual. So you have to swallow the pill and create a temporary file.
Original Answer:
This is just the way of temporary storing. I hope you manage the zip handling and parsing of xml on your own.
Use the php php://memory (doc) wrapper. Be aware, that this is only usefull for small files, because its stored in the memory - obviously. Otherwise use php://temp instead.
<?php
// the decoded content of your zip file
$text = 'base64 _decoded_ zip content';
// this will empty the memory and appen your zip content
$written = file_put_contents('php://memory', $text);
// bytes written to memory
var_dump($written);
// new instance of the ZipArchive
$zip = new ZipArchive;
// success of the archive reading
var_dump(true === $zip->open('php://memory'));
toster-cx had it right,you should award him the points, this is an example where the zip comes from a soap response as a byte array (binary), the content is an XML file:
$objResponse = $objClient->__soapCall("sendBill",array(parameters));
$fileData=unzipByteArray($objResponse->applicationResponse);
header("Content-type: text/xml");
echo $fileData;
function unzipByteArray($data){
/*this firts is a directory*/
$head = unpack("Vsig/vver/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr($data,0,30));
$filename = substr($data,30,$head['namelen']);
$if=30+$head['namelen']+$head['exlen']+$head['csize'];
/*this second is the actua file*/
$head = unpack("Vsig/vver/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr($data,$if,30));
$raw = gzinflate(substr($data,$if+$head['namelen']+$head['exlen']+30,$head['csize']));
/*you can create a loop and continue decompressing more files if the were*/
return $raw;
}
If you know the file name inside the .zip, just do this:
<?php
$xml = file_get_contents('zip://./your-zip.zip#your-file.xml');
If you have a plain string, just do this:
<?php
$xml = file_get_contents('compress.zlib://data://text/plain;base64,'.$base64_encoded_string);
[edit] Documentation is there: http://www.php.net/manual/en/wrappers.php
From the comments: if you don't have a base64 encoded string, you need to urlencode() it before using the data:// wrapper.
<?php
$xml = file_get_contents('compress.zlib://data://text/plain,'.urlencode($text));
[edit 2] Even if you already found a solution with a file, there's a solution (to test) I didn't see in your answer:
<?php
$zip = new ZipArchive;
$zip->open('data::text/plain,'.urlencode($base64_decoded_string));
$zip2 = new ZipArchive;
$zip2->open('data::text/plain;base64,'.urlencode($base64_string));
If you are running on Linux and have administration of the system. You could mount a small ramdisk using tmpfs, the standard file_get / put and ZipArchive functions will then work, except it does not write to disk, it writes to memory.
To have it permanently ready, the fstab is something like:
/media/ramdisk tmpfs nodev,nosuid,noexec,nodiratime,size=2M 0 0
Set your size and location accordingly so it suits you.
Using php to mount a ramdisk and remove it after using it (if it even has the privileges) is probably less efficient than just writing to disk, unless you have a massive number of files to process in one go.
Although this is not a pure php solution, nor is it portable.
You will still need to remove the "files" after use, or have the OS clean up old files.
They will of coarse not persist over reboots or remounts of the ramdisk.
if you want to read the content of a file from zip like and xml inside you shoud look at this i use it to count words from docx (wich is a zip )
if (!function_exists('docx_word_count')) {
function docx_word_count($filename)
{
$zip = new ZipArchive();
if ($zip->open($filename) === true) {
if (($index = $zip->locateName('docProps/app.xml')) !== false) {
$data = $zip->getFromIndex($index);
$zip->close();
$xml = new SimpleXMLElement($data);
return $xml->Words;
}
$zip->close();
}
return 0;
}
}
The idea comes from toster-cx is pretty useful to approach malformed zip files too!
I had one with missing data in the header, so I had to extract the central directory file header by using his method:
$CDFHoffset = strpos( $zipFile, "\x50\x4b\x01\x02" );
$CDFH = unpack( "Vsig/vverby/vverex/vflag/vmeth/vmodt/vmodd/Vcrc/Vcsize/Vsize/vnamelen/vexlen", substr( $zipFile, $CDFHoffset, 46 ) );
I have a processing server with my database and a serving database to serve up files with a low bandwidth cost. On the processing server, php is not able to create files so everything must be done with streams and/or stay in memory before being sent over to another server for download. A few days ago I found out about the stream abstraction with 'php://memory' and that I can do something like
$fp=fopen('php://memory','w+');
fwrite($fp,"Hello world");
fseek($fp,0,SEEK_SET);
//make a ftp connection here with $conn_id
$upload = ftp_fput($conn_id,"targetpath/helloworld.txt",$fp,FTP_BINARY);
to make the file in memory and then allow me to ftp it over to my other server. This is exactly what I want, except I also want to zip the data before sending it -- preferably using only native parts of php like ziparchive and not additional custom classes for special stream manipulation. I know that I am very close with the following...
$zip = new ZipArchive();
if($zip->open('php://memory', ZIPARCHIVE::CREATE)) {
$zip->addFromString('testtext.txt','Hello World!');
$fp = $zip->getStream('test'); if(!$fp) print "no filepointer";
//make a ftp connection here with $conn_id
$upload = ftp_fput($conn_id,"targetpath/helloworld.zip",$fp,FTP_BINARY);
} else print "couldn't open a zip like that";
The point at which this fails is the call to getStream (which always returns false although I think I am using correctly). It appears that the zip is fine making the file in 'php://memory' but for some reason getStream still fails although perhaps I don't sufficiently understand how ZipArchive makes zips...
How can I go from the string to the zipped filepointer so that I can ftp the zip over to my other server? Remember I can't make any files or else I would just make the zip file then ftp it over.
EDIT: based on skinnynerd's suggestions below I tried the following
$zip = new ZipArchive();
if($zip->open('php://memory', ZIPARCHIVE::CREATE)) {
$zip->addFromString('testtext.txt','Hello World!');
$zip->close();
$fp = fopen('php://memory','r+');
fseek($fp,0,SEEK_SET);
//connect to ftp
$upload = ftp_fput($conn_id,"upload/transfer/helloworld.zip",$fp,FTP_BINARY);
}
This does make a zip and send it over but the zip is 0 bytes large so I don't think that 'php://memory' works the way I thought... it actually fails at the close step -- the $zip->close() returns false which makes me wonder if I can open zips into 'php://memory' at all. Does anyone know what I can try along these line to get the zip?
$zip->getStream('test') is getting a stream to extract the file 'test' from the archive. Since there's no file 'test' in the archive, this fails. This is not the function you want to use.
As you said, what you want to do is send the finished archive to the ftp server. In this case, you would want to close the zip archive, and then reopen php://memory as a normal file (using fopen) to send it.
I don't know, but you may also be able to use $zip as a resource directly, without having to close and reopen the file.
And I think you can try create a stream pipe directly from ftp server
<?php
$zip = new ZipArchive();
if($zip->open('ftp://user:password#ftp.host.com/upload/transfer/helloworld.zip', ZipArchive::CREATE))
{
$zip->addFromString('testtext.txt','Hello World!');
$zip->close();
}
else
print "couldn't open zip file on remote ftp host.";
Does it have to be a Zip archive? Since you're trying to save bandwith it could be a gzip too.
<?php
$ftp_credentials = "ftp://USER:PASSWORD#HOST/helloworld.gz";
$gz = gzencode("Hello World!", 9);
$options = array('ftp' => array('overwrite' => true));
$stream_context = stream_context_create($options);
file_put_contents($ftp_credentials, $gz, 0, $stream_context);
?>
I need to read a list of CSV files from an FTP and delete them after I successfully read them.
Until now, i opened the csv file using fopen into a resource and then used fgetcsv to read the csv lines from it.
$res = fopen($url);
while ($csv_row = fgetcsv($res, null, self::DELIMITER)) {
.....
}
The problem is that I need to read a list of csv files and delete them too. the ftp_get function save the file into a local file. I rather avoid that. any way I can keep using the fgetcsv function with the ftp_nlist & ftp_connect functions? ?
You can save the csv file to a temporary file stream using ftp_fget(). This allows you to avoid the "create-read-delete" cycle. Once you close the file stream it's like it magically never existed :)
$ftp_handle = ftp_connect($ftp_server);
$remote_path = "/path/to/file.csv";
$tmp_handle = fopen('php://temp', 'r+');
if (ftp_fget($ftp_handle, $tmp_handle, $remote_path, FTP_ASCII)) {
rewind($tmp_handle);
while ($csv_row = fgetcsv($tmp_handle)) {
// do stuff
}
}
fclose($tmp_handle);
If you wanted to loop over a directory of files just get the list of files and then put the above code in a loop.