I need to read a large file to find some labels and create a dynamic form. I can not use file() or file_get_contents() because the file size.
If I read the file line by line with the following code
set_time_limit(0);
$handle = fopen($file, 'r');
set_time_limit(0);
if ($handle) {
while (!feof($handle)) {
$line = fgets($handle);
if ($line) {
//do something.
}
}
}
echo 'Read complete';
I get the following error in Chrome:
Error 101 (net::ERR_CONNECTION_RESET)
This error occurs after several minutes so that the constant max_input_time, I think not is the problem.(is set to 60).
What browser software do you use? Apache, nginx? You should set the max accepted file upload at somewhere higher than 500MB. Furthermore, the max upload size in the php.ini should be bigger than 500MB, too, and I think that PHP must be allowed to spawn processes larger than 500MB. (check this in your php config).
Set the memory limit ini_set("memory_limit","600M");also you need to set the time out limit
set_time_limit(0);
Generally long running processes should not be done while the users waits for them to complete.
I'd recommend using a background job oriented tool that can handle this type of work and can be queried about the status of the job (running/finished/error).
My first guess is that something in the middle breaks the connection because of a timeout. Whether it's a timeout in the web server (which PHP cannot know about) or some firewall, it doesn't really matter, PHP gets a signal to close the connection and the script stops running. You could circumvent this behaviour by using ignore-user-abort(true), this along with set_time_limit(0) should do the trick.
The caveat is that whatever caused the connection abort will still do it, though the script would still finish it's job. One very annoying side effect is that this script could possibly be executed multiple times in parallel without neither of them ever completing.
Again, I recommend using some background task to do it and an interface for the end-user (browser) to verify the status of that task. You could also implement a basic one yourself via cron jobs and database/text files that hold the status.
Related
I'm trying to develop an online management system for a very large FLAC music library for a radio station. It's got a beefy server and not many users, so I want to be able to offer a file download service where PHP transcodes the FLAC files into MP3/WAV depending on what the endpoint wants.
This works fine:
if($filetype == "wav") {
header("Content-Length: ". $bitrate * $audio->get_length());
$command = "flac -c -d ".$audio->get_filename().".flac";
}
ob_end_flush();
$handle = popen($command, "r");
while($read = fread($handle, 8192)) echo $read;
pclose($handle);
and allows the server to start sending the file to the user before the transcoding (well, decoding in this case) completes, for maximum speed.
However, the problem I'm getting is that while this script is executing, I can't get Apache to handle any other requests on the entire domain. It'll still work fine on other VirtualHosts on the same machine, but nobody can load any pages on this website while one person happens to be downloading a file.
I've also tried implementing the same thing using proc_open with no difference, and have played with the Apache settings for number of workers and the like.
Is the only way to stop this behaviour to use something like exec and waiting for the encoding process to finish before I start sending the user the file? Because that seems sub-optimal! :(
UPDATE: it seems that other people can still access the website, but not me - i.e. it's somehow related to sessions. This confuses me even more!
Use session_write_close() at some point before you start streaming... You may also want to stream_set_blocking(false) on the read pipe.
I write the php code in iis to serve file for download with speed limit, so i need to use sleep function for the speed limit.
Here, few lines of my code:
set_time_limit(0);
while(!feof($file))
{
echo fread($file, 1024*10);
ob_flush();
flush();
sleep(1);
if (connection_status()!=0)
{
#fclose($file);
exit;
}
}
But the browser say: 'Waiting for mysite'. If i remove sleep(1) everything is right. I also test in apache and everything is right too.
So I have a problem in IIS with the sleep function.
You need to have your server properly configured for that. TBH you should use something on the server to do that, rather then relying on PHP, the sleep(1); causes it to send a chunk, pause, send a chunk pause, etc. It does not maintain 10kbps but goes from like 500kbps for a second to 0 kbps for a second, it may average out to 10kbps, but it is not the same and some programs won't treat it correct and may terminate the download. You should look into QoS (How to Limit Download Speeds from my Website on my IIS Windows Server?)
What exactly is the problem with IIS? Note that waiting for 1 second will mean that your script may exceed the timeout limit (this can be as low as 30 seconds) so IIS will kill your script.
If you want to serve large files, I recommend serving them directly from IIS and using IIS' built-in rate limiter rather than via PHP.
See here: http://www.iis.net/configreference/system.applicationhost/sites/site/limits
I've written a simple PHP script to download a hidden file if the user has proper authentication. The whole set up works fine: it sends the proper headers, and the file transfer begins just fine (and ends just fine - for small files).
However, when I try to serve a 150 MB file, the connection gets mysteriously interrupted somewhere close to the middle of the file. Here's the relevant code fragment (taken from somewhere on the Internet and adapted by me):
function readfile_chunked($filename, $retbytes = TRUE) {
$handle = fopen($filename, 'rb');
if ($handle === false) return false;
while (!feof($handle) and (connection_status()==0)) {
print(fread($handle, 1024*1024));
set_time_limit(0);
ob_flush();
flush();
}
return fclose($handle);
}
I also do some other code BEFORE calling that function above, to try to solve the issue, but as far as I can tell, it does nothing:
session_write_close();
ob_end_clean();
ignore_user_abort();
set_time_limit(0);
As you can see, it doesn't attempt to load the whole file in memory at once or anything insane like that. To make it even more puzzling, the actual point in the transfer where it kills it seems to float between 50 and 110 MB, and it seems to kill ALL connections to the same file within a few seconds of each other (tried this by trying to download simultaneously with a friend). Nothing is appended to the interrupted file, and I see no errors on the logs.
I'm using Dreamhost, so I suspect that their watchdog might be killing my process because it's been running for too long. Does anyone have any experience to share on the matter? Could something else be the issue? Is there any workaround?
For the record, my Dreamhost is setup to use PHP 5.2.1 FastCGI.
I have little experience with Dreamhost, but you could use mod_xsendilfe instead (if Dreamhost allows it).
I have a backup script which backups up all files for a website to a zip file (using a script similar to the answer to this question). However, for large sites the script times out before it can complete.
Is there any way I can extend the length of time available for the script to run? The websites run on shared Windows servers, so I don't have access to the php.ini file.
If you are in a shared server environment, and you don’t have access to the php.ini file, or you want to set php parameters on a per-site basis, you can use the .htaccess file (when running on an Apache webserver).
For instance, in order to change the max_execution_time value, all you need to do is edit .htaccess (located in the root of your website, usually accesible by FTP), and add this line:
php_value max_execution_time 300
where 300 is the number of seconds you wish to set the maximum execution time for a php script.
There is also another way by using ini_set function in the php file
eg. TO set execution time as 5 second, you can use
ini_set('max_execution_time', 300); //300 seconds = 5 minutes
Please let me know if you need any more clarification.
set time limit comes to mind, but may still be limited by php.ini settings
set_time_limit(0);
http://php.net/manual/en/function.set-time-limit.php
Simply put; don't make a HTTP request to start the PHP script. The boundaries you're experiencing are set because you're using a HTTP request, which means you can have a time-out. A better solution would be to implement this using a "cronjob", or what Microsoft calls "Scheduled tasks". Most hosting providers will allow you to run such a task at set times. By calling the script from command line, you don't have to worry about the time-outs any more, but you're still at risk of running into memory issues.
If you have a decent hosting provider though, why doesn't it provide daily backups to start with? :)
You can use the following in the start of your script:
<?php
if(!ini_get('safe_mode')){
set_time_limit(0); //0 in seconds. So you set unlimited time
}
?>
And at the end of the script use flush() function to tell PHP to send out what it has generated.
Hope this solves your problem.
Is the script giving the "Maximum execution time of xx seconds exceeded" error message, or is it displaying a blank page? If so, ignore_user_abort might be what you're looking for. It tells php not to stop the script execution if the communication with the browser is lost, which may protect you from other timeout mechanisms involved in the communication.
Basically, I would do this at the beginning of your script:
set_time_limit(0);
ignore_user_abort(true);
This said, as advised by Berry Langerak, you shouldn't be using an HTTP call to run your backup. A cronjob is what you should be using. Along with a set_time_limit(0), it can run forever.
In shared hosting environments where a change to the max_execution_time directive might be disallowed, and where you probably don't have access to any kind of command line, I'm afraid there is no simple (and clean) solution to your problem, and the simplest solution is very often to use the backup solution provided by the hoster, if any.
Try the function:
set_time_limit(300);
On windows, there is a slight possibility that your webhost allows you to over ride settings by uploading a php.ini file in the root directory of your webserver. If so, upload a php.ini file containing:
max_execution_time = 300
To check if the settings work, do a phpinfo() and check the Local Value for max_execution_time.
Option 1: Ask the hosting company to place the backups somewhere accesible by php, so the php file can redirect the backup.
Option 2: Split the backup script in multiple parts, perhaps use some ajax to call the script a few times in a row, give the user a nice progress bar and combine the result of the script calls in a zip with php and offer that as a download.
I have an upload form that uploads mp3s to my site. I have some intermittent issues with some users which I suspect to be slow upload connections...
But anyway the first line of code is set_time_limit(0); which did fix it for SOME users that had connections that were taking a while to upload, but some are still getting timed out and I have no idea why.
It says the script has exceeded limit execution of 60 seconds. The script has no loops so it's not like it's some kind of infinite loop.
The weird thing is that no matter what line of code is in the first line it will always say "error on line one, two, etc" even if it's set_time_limit(0);. I tried erasing it and the very first line of code always seems to be the error, it doesn't even give me a hint of why it can't execute the php page.
This is an issue only few users are experiencing and no one else seems to be affected. Could anyone throw some ideas as to why this could be happening?
set_time_limt() will only effect the actual execution of the PHP code on the page. You want to set the PHP directive max_input_time, which controls how long the script will accept input (like files) for. The catch is that you need to set this in php.ini, as if the default max_input_time is exceeded, it'll never reach the script which is attempting to change it with ini_set().
Sure, a couple of things noted in the PHP Manual.
Make sure PHP is not running in safe-mode. set_time_limit has no affect when PHP is running in safe_mode.
Second, and this is where I assume your problem lies.....
Note: The set_time_limit() function and the configuration directive max_execution_time only affect the execution time of the script itself. Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real.
So your stream may be the culprit.
Can you post a little of your upload script, are you calling a separate file to handle the upload using Headers?
Try ini_set('max_execution_time', 0); instead.