Is it going to be a strain on my server if I am using PHP scripts to download a file while the file is very large, and it takes around 4 minutes to download it via PHP. Currently I'm executing the script in the browser, but when I switch to my linux server it will be executed via the shell. Right now I have put this in my script:
ini_set('max_execution_time', 5000);
Are there any negative factors to using ini_set I should be aware of when it takes quite a bit of time for a PHP script to execute because it is download a large .ZIP file? Should I have to worry about memory leaks?
Instead of just sending the file in one large chunk, why not split it up into lots of little chunks. The benefit of that is it stops php from timing out & lets you download larger files.
Take a look at this tutorial for more info http://teddy.fr/blog/how-serve-big-files-through-php
Related
I've got a backup script that continually builds excel files as it executes. The main script calls a class to handle the excel file build a few hundred times (one for each excel file).
The problem is that with each cycle of the loop, calling on the class to build the next excel file just adds to the used memory. Eventually this overwhelms the memory allocated for the execution.
I know the temporary answer is increase the memory allowed, but I was hoping I could wrap the file building with some memory-type ob_start/ob_clean functions.
I've tried to debug with xdebug for the past couple of days and I don't see any gaping holes that would cause the problem -- the memory usage is a pretty gradual increase overtime.
Thank you!
Try separating the actual "work" in a separate .php script and calling that using shell_exec (http://php.net/manual/en/function.shell-exec.php) from your main script.
That way any memory allocated by the "worker" script is automatically cleared when it finishes executing.
Also you might look at CRON jobs to execute the main script at intervals instead of allowing it to run as a daemon.
There's also forking.
How to create a process on the server like ftp_get() but not waiting its results to continue the PHP script?
My issue is I'm working on a synchronization script and some files are really huge to be downloaded using PHP since it conflicts with max execution time.
Is there any way to initiate the process to download the file and leave it to proccess another?
You need threading in PHP.
See http://php.net/manual/en/class.thread.php, if you don't have experience with threading then you should look up some tutorials and examples and then some. After thinking you understand them, research them some more.
And maybe a bit more...
Creating a multi-threaded application that is stable is a hard task.
Otherwise you could always increase the max execution time, or setup the cron job to download the FTP files in advance such as 30 minutes before with other linux utilities.
I'm generating about 700 PDFs with dompdf and saving them to the disk. Each one involves a database query to fetch the information for the PDF. When I run it I get about 2 minutes in and then it runs out of memory.
Why am I running out of memory? Shouldn't each PDF generation consume x memory, then release it when finished?
What's good practice in PHP for doing large operations like this to avoid strangling your server?
Some more info:
It generates many PDFs before running out of memory (some 100s)
The PDF it fails on is not abnormally large or special in any way. They're all between 4KB and ~500KB
Watching the memory usage as it processes it just slowly climbs in fits and starts until it runs out.
I can't get the wysiwyg to code-format properly, so here's a pastie for the snippets in question: http://pastie.org/3800751
Your problem probably that you running your code asynchronously. Try running it synchronously, it might take a really long time but it will work. Also, make sure to dispose of your objects at the end of each loop.
Ideas:
Increase memory limit ini_set
Use a cron job, or generally try and run each pdf generation
asynchronously with a queue.
Batch (split) processing into a number which
can be processed within one page load / Redirect after each batch to
same page using header / Keeping track of where you are on each script load with session / Beware of redirect limits on browsers
In general, large operations like this should be forked into child processes that each handle generating a single PDF. This should avoid the out of memory situation in case your PDF or DB libraries leak memory.
Try to change memory_limit at php.ini
I am trying to write a PHP script that will perform the following:
Extract all files from an archive in a given directory (using the exec function to actually run 7z)
List all the files that are images (jpg, jpeg, gif, png) and store them in an array (using exec again to run "ls" and storing only the images in the array, I compare the extension)
Cycle through the array and for each image:
Move that image in a unique folder
Generate thumbnails in that same unique folder (I use PHP Thumbnailer)
Record that new image in a database
For a small number of images, I would say that the script works ok (it could probably be improved a lot).
My problem is that after a certain number (which seems random each time I run my script) of treated images, I get a fatal error:
PHP Fatal error: Maximum execution time of 60 seconds exceeded in /var/www/ims/public_html/dev/include/phpThumb/GdThumb.inc.php on line 217
imagecopyresampled
(
$this->workingImage,
$this->oldImage,
0,
0,
0,
0,
$this->newDimensions['newWidth'],
$this->newDimensions['newHeight'],
$this->currentDimensions['width'],
$this->currentDimensions['height']
); // <- this is line 217
I am hoping that someone could point me towards a reason for that timeout. Is the GD library limited ?
Should I install a dedicated software on my linux server to specifically perform this task?
Thanks for your help.
The 60 second time limit is for the entire scripted process. It is primarily intended to keep a rogue php child process from tying up the apache child (or other web server resource) on a loaded server.
If you are using php as a shell scripting language rather than as a web application I would recommend either set_time_limit(0); or making sure you are running the script in the php cli interpreter which does this automatically.
If you are trying to perform a large batch operation in response to a document upload/form post within a web server, you might want to look into using some sort of job server to offload the processing work to instead of keeping the web server tied up for a long period of time. Gearman is one such system.
php limits the amout of time a script is allowed to execute. You can increase this in php.ini or on a per-script basis using set_time_limit
Go to your php.ini file and find max_execution_time (about 30% of the way down in mine). Change that number to what you want.
Don't forget to output an indicator that your script is running as you expect so you can see that things are happening. Also, this tends to make my CPU temperature climb pretty quickly, indicating there's probably a better way to do things.
I am trying to read large files lets say illustrator file or photoshop file using cron job in my system.
Files size varies from 20 mb - 300 mb
I have been using some function but it break in middle while reading. So i wanted to have a fresh opinion.
Amount these function
file_get_contents
readfile
curl
which is most effective in terms of
consistency (should not break while reading file)
speed
resource uses
if there is more then two cron job, does it impact over all server performance.
Please share best practice code.
Thanks in advance
Use cURL. The file functions have been deprecated in favor of cURL to open remote files. It's not only faster, but also more reliable1 (you are less likely to experience timeouts).
If your script times out or runs out of memory anyways, you'll want to increase the execution time and memory limits (max_execution_time and memory_limit).
Other notes:
readfile() reads a file and prints it to the output buffer; it's no the same thing as file_get_contents().
If you compile curl with --with-curlwrappers then when you do file_get_contents() it will use cURL instead of the fopen() functions.
1 Citation needed.
you need to split the two tasks if files are so big.
first you download the file with wget and once you have your file you process it with php.
this way you are less likely to go into timeout problems.
if you don't know which file to download because it is a variable from php of some sort you can write to a file the name of the required file as first step of your job
then pass it to wget via --input-file=file as second step
and then process it as third and final step with your php program
DirectIO is a low level extension that bypasses the OS and goes straight to the hard disk, as a result it is probably the most efficient.
http://php.net/manual/en/ref.dio.php
Note that as of PHP 5.1.0 it is no longer bundled with PHP. Also, if your script is breaking in the middle of the operation, check your max_execution_time and max_memory.