Generation of zip file in php is slow - php

I have a script which put images, floor plans and video into a zip file, it can reach 500mb easily but most of the time average is 150mb.
The generation of the zip file is extremely slow and i can't figurate why. Is there any tips to improve my script?
It took me 10 min to create the zip file in the server just for 100mb.
if( !empty( $files ) ){
$random_nbr = mt_rand(1,5646866662);
$path = 'webroot/img/tmp/' . $random_nbr;
if (!file_exists(\Cake\Core\Configure::read('pathTo') . 'webroot/img/tmp')) {
mkdir(\Cake\Core\Configure::read('pathTo') . 'webroot/img/tmp', 0777, true);
}
$destination = \Cake\Core\Configure::read('pathTo') . $path . '_media.zip';
$media_url = \Cake\Core\Configure::read('websiteUrl') . '/img/tmp/' . $random_nbr . '_media.zip';
$zip = new ZipArchive();
$zip->open( $destination, ZIPARCHIVE::CREATE | ZIPARCHIVE::OVERWRITE );
// Photos
if (isset($files['photos'])):
foreach( $files['photos'] as $f ){
$context = stream_context_create(array('http' => array('header'=>'Connection: close\r\n')));
// Original
$parsed_file = $f['original_file'];
$download_file = file_get_contents($parsed_file, false,$context);
$zip->addFromString('photos/original/' . basename($parsed_file), $download_file);
// Web with or without a watermark
$web = $this->Images->state_image(1270, $f['id'], 0, '');
$web = $web->response('jpg');
$zip->addFromString('photos/web/' . $f['name'], $web);
// High Res Web with or without a watermark
$web = $this->Images->state_image(2000, $f['id'], 0, '');
$web = $web->response('jpg');
$zip->addFromString('photos/high_res_web/' . $f['name'], $web);
}
endif;
// Floor Plan
if (isset($files['floorplan'])):
foreach( $files['floorplan'] as $f ){
$parsed_file = $f['original_file'];
$context = stream_context_create(array('http' => array('header'=>'Connection: close\r\n')));
$download_file = file_get_contents($parsed_file, false,$context);
$zip->addFromString('floorplan/' . basename($parsed_file), $download_file);
}
endif;
// Video
if (isset($files['video'])):
foreach( $files['video'] as $f ){
$parsed_file = $f['original_file'];
$context = stream_context_create(array('http' => array('header'=>'Connection: close\r\n')));
$download_file = file_get_contents($parsed_file, false,$context);
$zip->addFromString('floorplan/' . basename($parsed_file), $download_file);
}
endif;
$zip->close();
echo $media_url;
die();
}

Outside of dedicated hardware, there probably is not much you'll be able to do to speed up the actual zipping process. You could try exec()'ing the system zip utility rather than using PHP to do it, but that may not change things.
What you can do though (if the host allows it) is background the process and provide a status page so users can see how long until their file is ready. I've done this in the past for similar problems.
What I did was have a table in the database that would store information about the zip file to be created, and a list of all the files to be added to the zip file. Then I'd exec() off a background script with the ID of the newly created DB record.
The background process would read the DB for all the details and begin creating the zip file. Periodically it would update the DB with a % complete. When finished it'd update the DB with the file system path to the newly generated zip file.
Then I had another page for the end user that displayed a progress bar. The page would periodically make an Ajax request to the server to get the new % complete for the file and update the bar accordingly. When the file was complete it would change to a download link for them to begin downloading the file.
There was another cron job process that would periodically go through and delete all the temp files older than 5 days. If users needed the file again they had to have it re-generated.

Related

Mysql file upload and pdf browser viewer failing to work

I am trying to build an interface with between my Wordpress page and my practice management software. When I upload files directly to my practice management software a physical copy shows up in the patient file. When browsing files within the software, on first click, I am able to view them in a browser based pdf viewer. If I click again on the file link, the file downloads and opens on my PC's PDF software.
PROBLEM: currently my file uploads to the server and a physical file is placed in the patient file. When browsing files it will not show up in my PDF viewer. It downloads to my PC on the first click and opens into my PDF software. However, the same file, when uploaded directly to the software behaves as expected.
I see no difference in the two files. So I assume there must be a problem with the $contents variable in my query. I am only working with PDF files. To upload attachments in my private messaging software I am using the following code:
function action_getmsgup($uploadid) {
global $wpdb, $out;
$query = $wpdb->prepare("SELECT ID, post_mime_type, guid FROM {$wpdb->prefix}posts WHERE ID = %d", array($uploadid));
$rows = $wpdb->get_results($query, ARRAY_A);
foreach ($rows as $row) {
$url = $row['guid'];
$filename = basename($url);
$path = parse_url($url, PHP_URL_PATH); // just the path part of the URL
$parts = explode('/', $path); // all the components
$parts = array_slice($parts, -6); // the last six
$path = implode('/', $parts);
$filepath = ABSPATH . $path;
// Get file contents and make a blob.
$tmpfile = fopen($filepath, "r");
$contents = fread($tmpfile, filesize($filepath));
$out['filename'] = $filename;
$out['mimetype'] = $row['mimetype'];
$out['contents'] = $contents;
}
}
QUESTION: Is there a problem with my upload method that is not filling $contents correctly?

cakephp 3 lock file property

I have an application that create many php process to read file.
Sometimes process read the same file so I need to lock file when a process read It.
The application need to read only one times a file and after delete It.
Sometimes two process read the same file and this is a big problem.
I have create many process that every one loop through a folder, read all file, and delete it.
I don't want that two process read the same file, but the problem is that the process works simultaneously.
code of every process :
public function readAllXmlAndSave()
$xml_files = glob(TMP . 'xml/*.xml');
foreach($xml_files as $fileXml){
$explStr = explode('/', $fileXml);
$filename = $explStr[count($explStr) - 1];
$path = TMP . '/xml/' . $filename;
$file = new File($path, false);
if($file->exists()){
$string = $file->read();
//some work
$file->close();
$file->delete();
}
}
}
I have create a simple script to understand how lock file works into cakephp 3 but seems that property lock doesn't work as I aspected.
code
$path = TMP . '/xml/test.xml'; //inside there is "HELLO"
$file = new File($path, false);
$string = $file->read();
echo($string);
$file->lock = true;
$string2 = $file->read();
echo($string2);
$string3 = $file->read();
echo($string3);
I aspected that only the first echo work not the other.. Why I get always:
HELLO
HELLO
HELLO
and not only the first echo?
HELLO

On creating zip file by php I get two files instead of one

I'm struggling around with a simple PHP functionality: Creating a ZIP Archive with some files in.
The problem is, it does not create only one file called filename.zip but two files called filename.zip.a07600 and filename.zip.b07600. Pls. see the following screenshot:
The two files are perfect in size and I even can rename each of them to filename.zip and extract it without any problems.
Can anybody tell me what is going wrong???
function zipFilesAndDownload_Defect($archive_file_name, $archiveDir, $file_path = array(), $files_array = array()) {
// Archive File Name
$archive_file = $archiveDir."/".$archive_file_name;
// Time-to-live
$archiveTTL = 86400; // 1 day
// Delete old zip file
#unlink($archive_file);
// Create the object
$zip = new ZipArchive();
// Create the file and throw the error if unsuccessful
if ($zip->open($archive_file, ZIPARCHIVE::CREATE) !== TRUE) {
$response->res = "Cannot open '$archive_file'";
return $response;
}
// Add each file of $file_name array to archive
$i = 0;
foreach($files_array as $value){
$expl = explode("/", $value);
$file = $expl[(count($expl)-1)];
$path_file = $file_path[$i] . "/" . $file;
$size = round((filesize ($path_file) / 1024), 0);
if(file_exists($path_file)){
$zip->addFile($path_file, $file);
}
$i++;
}
$zip->close();
// Then send the headers to redirect to the ZIP file
header("HTTP/1.1 303 See Other"); // 303 is technically correct for this type of redirect
header("Location: $archive_file");
exit;
}
The code which calls the function is a file with a switch-case... it is called itself by an ajax-call:
case "zdl":
$files_array = array();
$file_path = array();
foreach ($dbh->query("select GUID, DIRECTORY, BASENAME, ELEMENTID from SMDMS where ELEMENTID = ".$osguid." and PROJECTID = ".$osproject.";") as $subrow) {
$archive_file_name = $subrow['ELEMENTID'].".zip";
$archiveDir = "../".$subrow['DIRECTORY'];
$files_array[] = $archiveDir.DIR_SEPARATOR.$subrow['BASENAME'];
$file_path[] = $archiveDir;
}
zipFilesAndDownload_Defect($archive_file_name, $archiveDir, $file_path, $files_array);
break;
One more code... I tried to rename the latest 123456.zip.a01234 file to 123456.zip and then unlink the old 123456.zip.a01234 (and all prior added .a01234 files) with this function:
function zip_file_exists($pathfile){
$arr = array();
$dir = dirname($pathfile);
$renamed = 0;
foreach(glob($pathfile.'.*') as $file) {
$path_parts = pathinfo($file);
$dirname = $path_parts['dirname'];
$basename = $path_parts['basename'];
$extension = $path_parts['extension'];
$filename = $path_parts['filename'];
if($renamed == 0){
$old_name = $file;
$new_name = str_replace(".".$extension, "", $file);
#copy($old_name, $new_name);
#unlink($old_name);
$renamed = 1;
//file_put_contents($dir."/test.txt", "old_name: ".$old_name." - new_name: ".$new_name." - dirname: ".$dirname." - basename: ".$basename." - extension: ".$extension." - filename: ".$filename." - test: ".$test);
}else{
#unlink($file);
}
}
}
In short: copy works, rename didn't work and "unlink"-doesn't work at all... I'm out of ideas now... :(
ONE MORE TRY: I placed the output of $zip->getStatusString() in a variable and wrote it to a log file... the log entry it produced is: Renaming temporary file failed: No such file or directory.
But as you can see in the graphic above the file 43051221.zip.a07200 is located in the directory where the zip-lib opens it temporarily.
Thank you in advance for your help!
So, after struggling around for days... It was so simple:
Actually I work ONLY on *nix Servers so in my scripts I created the folders dynamically with 0777 Perms. I didn't know that IIS doesn't accept this permissions format at all!
So I remoted to the server, right clicked on the folder Documents (the hierarchically most upper folder of all dynamically added files and folders) and gave full control to all users I found.
Now it works perfect!!! The only thing that would be interesting now is: is this dangerous of any reason???
Thanks for your good will answers...
My suspicion is that your script is hitting the PHP script timeout. PHP zip creates a temporary file to zip in to where the filename is yourfilename.zip.some_random_number. This file is renamed to yourfilename.zip when the zip file is closed. If the script times out it will probably just get left there.
Try reducing the number of files to zip, or increasing the script timeout with set_time_limit()
http://php.net/manual/en/function.set-time-limit.php

sugarCRM add document from script

I am trying to add Document to Sugar object (client) from PHP script. I have a directory of files (on the same server where sugarCRM is installed) and xls with sugar objec ID and filename). PHP Script should add correct filename to specific sugar object (identified with ID). I can read XLS this is no problem, I can also get instance of sugar object (retrieve by ID), but I have no idea how can I assign the file to sugar. I was trying with Document, and upload_file.php, but those seem to be usable with uploading single file with html Form.
How can I automate this task, copy files with correct filename to cache\upload and create Document related to my Customer from PHP Script? I would prefer not to use SOAP if it's not necesarry...
Edit:
I was able to save document and revision, but something is wrong, and file can't be downloaded from browser ("incorrect call to file")
My Code so far is:
require_once('include/upload_file.php');
$upload_file = new UploadFile('uploadfile');
$document->filename = 'robots.txt';
$document->document_name = 'robots.txt';
$document->save();
$contents = file_get_contents ($document->filename);
$revision = new DocumentRevision;
$revision->document_id = $document->id;
$revision->file = base64_encode($contents);
$revision->filename = $document->filename;
$revision->revision = 1;
$revision->doc_type = 'Sugar';
$revision->file_mime_type = 'text/plain';
$revision->save();
$document->revision_id = $revision->id;
$document->save();
$destination = clean_path($upload_file->get_upload_path($document->id));
$fp = sugar_fopen($destination, 'wb');
if( !fwrite($fp, $contents) ){
die("ERROR: can't save file to $destination");
}
fclose($fp);
WORKS! I Hope this will help someone
I have corrected 3 lines from code abowe:
//$document->revision_id = $revision->id;
//$document->save();
$destination = clean_path($upload_file->get_upload_path($revision->id));

Improve my Unzip & Move function - PHP

I'm a PHP novice and so looking for some advice on a PHP function i have created to use within a Wordpress installation.
As you can see from the code below, it runs when one of the admin's press 'Publish' on a pending post.
It takes a Zip file that has been uploaded by a user via Gravity Forms, then unzips ONLY .mp3 extensions. Re-zips and moves all the files to a new folder in our Amazon S3 directory.
The code is pieced together from my limited knowledge and some help along the way with questions on here.
So, here's what i ended up with:
add_action('pending_to_publish', 'unzip_to_s3');
function unzip_to_s3() {
global $post;
global $wpdb;
// Only run function if post is portfolio post type
if ('portfolio' == $post->post_type) {
// Set temp path
$temp_path = '../wp-content/uploads/gravity_forms/1-9e5dc27086c8b2fd2e48678e1f54f98c/2013/02/tmp/';
// Get filename from Zip file
$file = get_post_meta($post->ID, 'file_url', true);
$zip_file = basename($file);
// Create full Zip file path
$zip_file_path = $temp_path.$zip_file;
// Generate unique name for temp sub_folder for unzipped files
$temp_unzip_folder = uniqid('temp_TMS_', true);
// Create full temp sub_folder path
$temp_unzip_path = $temp_path.$temp_unzip_folder;
// Make the new temp sub_folder for unzipped files
if (!mkdir($temp_unzip_path, 0755, true)) {
die('Error: Could not create path: '.$temp_unzip_path);
}
// Unzip files to temp unzip folder, ignoring anything that is not a .mp3 extension
$zip = new ZipArchive();
$filename = $zip_file_path;
if ($zip->open($filename)!==TRUE) {
exit("cannot open <$filename>\n");
}
for ($i=0; $i<$zip->numFiles;$i++) {
$info = $zip->statIndex($i);
$file = pathinfo($info['name']);
if(strtolower($file['extension']) == "mp3") {
file_put_contents($temp_unzip_path.'/'.basename($info['name']), $zip->getFromIndex($i));
} else {
$zip->deleteIndex($i);
}
}
$zip->close();
// Re-zip the unzipped mp3's and store new zip file in temp folder created earlier
$temp_unzip_path = $temp_unzip_path.'/';
$zip = new ZipArchive();
$dirArray = array();
$new_zip_file = $temp_unzip_path.$zip_file;
$new = $zip->open($new_zip_file, ZIPARCHIVE::CREATE);
if ($new === true) {
$handle = opendir($temp_unzip_path);
while (false !== ($entry = readdir($handle))) {
if(!in_array($entry,array('.','..')))
{
$dirArray[] = $entry;
$zip->addFile($temp_unzip_path.$entry,$entry);
}
}
closedir($handle);
} else {
echo 'Failed to create Zip';
}
$zip->close();
// Set Media bucket dir
$bucket_path = '../wp-content/uploads/gravity_forms/1-9e5dc27086c8b2fd2e48678e1f54f98c/2013/02/mixtape2/';
// Generate unique name for sub_bucket
$sub_bucket = uniqid('TMS_', true);
// Create full sub_bucket path
$sub_bucket_path = $bucket_path.$sub_bucket;
// Make the new sub_bucket
if (!mkdir($sub_bucket_path, 0755, true)) {
die('Error: Could not create path: '.$sub_bucket_path);
}
// Move mp3's to new sub_bucket
// Get array of all source files
$files = scandir($temp_unzip_path);
// Identify directories
$source = $temp_unzip_path;
$destination = $sub_bucket_path.'/';
// Cycle through all source files
foreach ($files as $file) {
if (in_array($file, array(".",".."))) continue;
// if move files is successful delete the original temp folder
if (rename($source.$file, $destination.$file)) {
rmdir($temp_unzip_path);
}
}
// Delete original Zip file
unlink($temp_path.$zip_file);
// Update Custom field for new Zip file location
update_post_meta($post->ID, 'file_url', 'http://themixtapesite.com/wp-content/uploads/gravity_forms/1-9e5dc27086c8b2fd2e48678e1f54f98c/2013/02/mixtape2/'.$sub_bucket.'/'.$zip_file);
}
}
Whilst this function does work, we're dealing with large files and so it does take a while to process...
What is happening is when the admin presses publish it triggers this function but the page just sits there until it's finished this function and then will continue. This function can take upto around 5 minutes to run.
I'm looking to optimise this function (in terms of code) but also see if there's a way i can run this in the background so that the admin can carry on with other things and not have to sit there waiting around.
Any help appreciated.
You may want to try to WP cron and schedule the task at that point so that it runs in the background. Here is some resources for that. the basic concept would go something like this.
if ( ! wp_next_scheduled( 'pending_to_publish' ) ) {
wp_schedule_single_event($timestamp,'pending_to_publish');
}
add_action('pending_to_publish', 'unzip_to_s3');
http://wpengineer.com/1908/use-wordpress-cron/
http://codex.wordpress.org/Category%3aWP-Cron_Functions
https://wordpress.stackexchange.com/questions/42694/run-a-cron-job-or-similar-in-the-background-of-wp-after-post-update-create

Categories