Zip thousand files in php - memory leak - php

I have to zip search results containing maximum 10000 files, with an approximate dimension of far more than 1Gb.
I create a zip archive and read every file in a for loop whit fread and add the resulting file to the archive.
I never finished adding files because of this error
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 1257723 bytes)
but I don't think adding 1Gb or more to the memory_limit value of php.ini could be a solution, because memory resources are limited.
Because of the zip file will stay in memory until it will be closed (or so I read in another question), I made a way to create a series of zip file of 50Mb to avoid the memory leak. But even if the php script create another file zip, it will stop with the same PHP Fatal error on the same file (the 174th).
Why?
Am I doing something wrong?
Any help will be appreciated.
Here is a code snippet of the file creation
$zip = new ZipArchive();
$nomeZipFile = "../tmp/" . $title . ".zip";
for ($x = 0; $x < count($risultato); $x++) {
$numFiles = 0;
$dir = '../tmp';
if (file_exists($nomeZipFile)) {
if (filesize($nomeZipFile) > 52428800) {
// filename count
if ($handle = opendir($dir)) {
while (($file = readdir($handle)) !== false) {
if (!in_array($file, array('.', '..')) && !is_dir($dir . $file))
$numFiles++;
}
}
$nomeZipFile = '../tmp/' . $title . $numFiles . '.zip';
$res = $zip->open($nomeZipFile, ZIPARCHIVE::CREATE);
} else {
$res = $zip->open($nomeZipFile, ZIPARCHIVE::CREATE);
}
} else {
$res = $zip->open($nomeZipFile, ZIPARCHIVE::CREATE);
}
...
// adding file
$fileDownload = "";
$fDownload = fopen($kt_response->message, "r"); // the file is downloaded though a webservice
while(!feof($fDownload)) { $fileDownload .= fread($fDownload, 1024); flush(); }
$zip->addFromString(filename, $fileDownload);
....
$zip->close();
}

Related

Why is this zip function ignoring/skipping files?

I have built an application which uploads HTML canvas elements to a directory on the server, and then zips and downloads that directory.
The first part of this (uploading the canvas elements) is working perfectly. All of the .png files are uploaded 100% of the time. However, whenever I come to zip and download the directory contents, it seems that files are being missed/skipped.
Here is the code I'm using to zip the directory:
if($x == $l) { // $x = number of .PNGs, $l = expected number
$zip = new ZipArchive();
$zipPath = "../titles/" . $folder . ".zip";
if ($zip->open($zipPath, ZIPARCHIVE::CREATE )!==TRUE) {
exit("cannot open" . $zipPath> . "\n");
}
$list = scandir($path);
foreach($list as $value) {
$ex = explode(".", $value);
if($ex[1]=="png") {
$zip->addFile($path.$value, $value);
}
}
$zip->close();
echo "titles/" . $folder . ".zip"; // Sends path back to browser for download
}
There is no consistent pattern with files being missed out of the zip folder. Occasionally they will all be there, but more often than not some will be missing.
Does anyone know what might be causing this/how it could be fixed? Thanks!

CakePHP making Zip files, getting file 'bytes exhausted' error message?

I am working with Files and Folders within CakePHP. Now everything works fine and in the way I want it to. However, when Zipping files, I get the following error message :
Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 240047685 bytes)
Now zipping smaller files, its fine! I have even done files in size of around 10MB without any issues, however zipping that are larger in size seem to have an issue.
Now I have added the following to my .htaccess file and made a php.ini file as I thought that might be the issue.
php_value upload_max_filesize 640000000M
php_value post_max_size 640000000M
php_value max_execution_time 30000000
php_value max_input_time 30000000
Until I found some posts pointing at the fact that PHP as a 4GB file limit. Well even if that is the case, why does my zip file not do this file (which is only about 245mb).
public function ZippingMyData() {
$UserStartPath = '/data-files/tmp/';
$MyFileData = $this->data['ZipData']; //this is the files selected from a form!
foreach($MyFileData as $DataKey => $DataValue) {
$files = array($UserStartPath.$DataValue);
$zipname = 'file.zip';
$zip = new ZipArchive();
$zip_name = time().".zip"; // Zip name
$zip->open($zip_name, ZipArchive::CREATE);
foreach ($files as $file) {
$path = $file;
if(file_exists($path)) {
$zip->addFromString(basename($path), file_get_contents($path));
} else {
echo"file does not exist";
}
} //End of foreach loop for $files
} //End of foreach for $myfiledata
$this->set('ZipName', $zip_name);
$this->set('ZipFiles', $MyFileData);
$zip->close();
copy($zip_name,$UserStartPath.$zip_name);
unlink($zip_name); //After copy, remove temp file.
$this->render('/Pages/download');
} //End of function
Any ideas of where I am going wrong? I will state that this is NOT my code, I found bits of it on others posts and changed it to fit my needs for my project!
All help most welcome...
Thanks
Glenn.
I think that ZipArchive loads your file in memory, so you have to increase the memory_limit parameter in php.ini.
To avoid consuming all the memory of your server and drop performance, if your file is big, a better (but far from be the best) solution should be:
public function ZippingMyData() {
$UserStartPath = '/data-files/tmp/';
$MyFileData = $this->data['ZipData']; //this is the files selected from a form!
foreach($MyFileData as $DataKey => $DataValue) {
$files = array($UserStartPath.$DataValue);
$zip_name = time().".zip"; // Zip name
// Instead of a foreach you can put all the files in a single command:
// /usr/bin/zip $UserStartPath$zip_name $files[0] $files[1] and so on
foreach ($files as $file) {
$path = $file;
if(file_exists($path)) {
exec("/usr/bin/zip $UserStartPath$zip_name basename($path)");
} else {
echo"file does not exist";
}
} //End of foreach loop for $files
} //End of foreach for $myfiledata
$this->render('/Pages/download');
} //End of function
or similar (depends on your server). This solution has only two limits: disk space and zip limitations.
I apologize for the poor quality of my code and for any error.

Unzipping a file with PHP, but not all files are extracted

I'm working on extracting a zip archive with PHP. The structure of the archive is seven folders, each of which contains on the order of 10,000 files, each around 1 kB.
My code is pretty simple and uses the ZipArchive class:
$zip = new ZipArchive();
$result = $zip->open($filename);
if ($result === true) {
$zip->extractTo($tmpdir);
$zip->close();
}
The problem I'm having, though, is that the extraction seems to halt. The first folder is fully extracted, but only about half of the second one is. None of the other five are extracted at all.
I also tried using this code, which breaks it into chunks of 10 kB at a time, but got the exact same result:
$archive = zip_open($filename);
while ($entry = zip_read($archive)) {
$size = zip_entry_filesize($entry);
$name = zip_entry_name($entry);
if (substr($name, -1) == '/') {
if (!file_exists($tmpdir . $name)) mkdir($tmpdir . $name);
} else {
$unzipped = fopen($tmpdir . $name, 'wb');
while ($size > 0) {
$chunkSize = ($size > 10240) ? 10240 : $size;
$size -= $chunkSize;
$chunk = zip_entry_read($entry, $chunkSize);
if ($chunk !== false) fwrite($unzipped, $chunk);
}
fclose($unzipped);
}
}
I've also tried increasing the memory limit in PHP from 512 MB to 1024 MB, but again got the same result. Unzipped everything is around 100 MB, so I wouldn't anticipate it being a memory issue anyway.
Probably its your max execution time... disable the limit completely by setting it to 0 or set a good value.
ini_set('max_execution_time', 10000);
... dont set it to 0 in production use ...
If you don't have access to ini_set() because of the disable_function directive you may have to edit its value in your php.ini directly.

Split a large zip file in small chunks using php script

I am using below script for spliting a large zip file in small chucks.
$filename = "pro.zip";
$targetfolder = '/tmp';
// File size in Mb per piece/split.
// For a 200Mb file if piecesize=10 it will create twenty 10Mb files
$piecesize = 10; // splitted file size in MB
$buffer = 1024;
$piece = 1048576*$piecesize;
$current = 0;
$splitnum = 1;
if(!file_exists($targetfolder)) {
if(mkdir($targetfolder)) {
echo "Created target folder $targetfolder".br();
}
}
if(!$handle = fopen($filename, "rb")) {
die("Unable to open $filename for read! Make sure you edited filesplit.php correctly!".br());
}
$base_filename = basename($filename);
$piece_name = $targetfolder.'/'.$base_filename.'.'.str_pad($splitnum, 3, "0", STR_PAD_LEFT);
if(!$fw = fopen($piece_name,"w")) {
die("Unable to open $piece_name for write. Make sure target folder is writeable.".br());
}
echo "Splitting $base_filename into $piecesize Mb files ".br()."(last piece may be smaller in size)".br();
echo "Writing $piece_name...".br();
while (!feof($handle) and $splitnum < 999) {
if($current < $piece) {
if($content = fread($handle, $buffer)) {
if(fwrite($fw, $content)) {
$current += $buffer;
} else {
die("filesplit.php is unable to write to target folder");
}
}
} else {
fclose($fw);
$current = 0;
$splitnum++;
$piece_name = $targetfolder.'/'.$base_filename.'.'.str_pad($splitnum, 3, "0", STR_PAD_LEFT);
echo "Writing $piece_name...".br();
$fw = fopen($piece_name,"w");
}
}
fclose($fw);
fclose($handle);
echo "Done! ".br();
exit;
function br() {
return (!empty($_SERVER['SERVER_SOFTWARE']))?'<br>':"\n";
}
?>
But this script not creating small files after split in target temp folder. Script runs successfully without any error.
Please help me to found out what is issue here? Or If you have any other working script for similar functinality, Please provide me.
As indicated in the comments above, you can use split to split a file into smaller pieces, and can then use cat to join them back together.
split -b50m filename x
and to put them back
cat xaa xab xac > filename
If you are looking to split the zipfile into a spanning type archive, so that you do not need to rejoin the them together take a look at zipsplit
zipslit -n (size) filename
so you can just call zipsplit from your exec script and then most standard unzip utils should be able to put it back together. man zipslit for more options, including setting output path, etc..

Unzipping larger files with PHP

I'm trying to unzip a 14MB archive with PHP with code like this:
$zip = zip_open("c:\kosmas.zip");
while ($zip_entry = zip_read($zip)) {
$fp = fopen("c:/unzip/import.xml", "w");
if (zip_entry_open($zip, $zip_entry, "r")) {
$buf = zip_entry_read($zip_entry, zip_entry_filesize($zip_entry));
fwrite($fp,"$buf");
zip_entry_close($zip_entry);
fclose($fp);
break;
}
zip_close($zip);
}
It fails on my localhost with 128MB memory limit with the classic "Allowed memory size of blablabla bytes exhausted". On the server, I've got 16MB limit, is there a better way to do this so that I could fit into this limit? I don't see why this has to allocate more than 128MB of memory. Thanks in advance.
Solution:
I started reading the files in 10Kb chunks, problem solved with peak memory usage arnoud 1.5MB.
$filename = 'c:\kosmas.zip';
$archive = zip_open($filename);
while($entry = zip_read($archive)){
$size = zip_entry_filesize($entry);
$name = zip_entry_name($entry);
$unzipped = fopen('c:/unzip/'.$name,'wb');
while($size > 0){
$chunkSize = ($size > 10240) ? 10240 : $size;
$size -= $chunkSize;
$chunk = zip_entry_read($entry, $chunkSize);
if($chunk !== false) fwrite($unzipped, $chunk);
}
fclose($unzipped);
}
Why do you read the whole file at once?
$buf = zip_entry_read($zip_entry, zip_entry_filesize($zip_entry));
fwrite($fp,"$buf");
Try to read small chunks of it and writing them to a file.
Just because a zip is less than PHP's memory limit & perhaps the unzipped is as well, doesn't take account of PHP's overhead generally and more importantly the memory needed to actually unzip the file, which whilst I'm not expert with compression I'd expect may well be a lot more than the final unzipped size.
For a file of that size, perhaps it is better if you use shell_exec() instead:
shell_exec('unzip archive.zip -d /destination_path');
PHP must not be running in safe mode and you must have access to both shell_exec and unzip for this method to work.
Update:
Given that command line tools are not available, all I can think of is to create a script and send the file to a remote server where command line tools are available, extract the file and download the contents.
function my_unzip($full_pathname){
$unzipped_content = '';
$zd = gzopen($full_pathname, "r");
while ($zip_file = gzread($zd, 10000000)){
$unzipped_content.= $zip_file;
}
gzclose($zd);
return $unzipped_content;
}

Categories