I'm running really long task in php. It's a website crawler and It has to be polite and sleep for 5 seconds each page to prevent server overloading. Script starts with:
ignore_user_abort(1);
session_write_close();
ob_end_clean();
while (#ob_end_flush());
set_time_limit(0);
ini_set('max_execution_time',0);
After few hours (between 3-7h) script dies without any visible reason.
I've checked
apache error log (nothing)
php_errors.log (nothing)
output for errors (10 578 467b of debug output, no errors)
memory consumption (stable, around 3M from memory_get_usage(true) checked every 5 sec, limit set to 512M)
It's not browser, cause I was using wget and chrome to check with the similar reason.
Output is sent to browser every 2-3 seconds, so I don't think that's the fault + I ignore user abort.
Is there any other place I can check to find the issue?
I think there's a problem in the rest of your script, not Apache.
Try profiling your application using the register_tick_function with a light profiler like this and logging the memory usage, may be that.
Related
We've inherited a platform that has a crobjob that every minute curls a local php script three times with different parameters (curl -s -o --url https://localhost/myscript.php?option=XYZ -k). That script runs for about 1 minute and its possible multiple instances with the same option overlap for a bit of time. The script logs in a different file per option given and each log starts with the timestamp when the script started so it acts as an instance identifier.
The script has this skeleton:
<?php
$option=XYZ;
$scriptId = time();
$file = "log_$option.txt";
file_put_contents($file,"\n$scriptId: Start\n",FILE_APPEND);
session_start();
$expires = time()+60;
file_put_contents($file,"\n$scriptId: Expires at $expires\n",FILE_APPEND);
while(time()<$expires){
file_put_contents($file,"\n$scriptId: Not expired at ".time()."\n",FILE_APPEND);
switch($option){
case X:
do_db_stuff();
break;
...
}
file_put_contents($file,"\n$scriptId: Will sleep at ".time()."\n",FILE_APPEND);
sleep(13);
file_put_contents($file,"\n$scriptId: Woke up at ".time()."\n",FILE_APPEND);
}
file_put_contents($file,"\n$scriptId: Finished at ".time()."\n",FILE_APPEND);
Normally this script runs fine (even if they overlap when instance A is sleeping for the last time and instance B starts) but on occasion we have two issues that we can confirm with the logs:
sometimes it sleeps for less than 13 seconds (a
variable amount of time and always less than 13);
sometimes the script stops (no more logging after the "Will sleep" one, and we can verify that no db stuff is being done). [Update on this in Edit 2]
We've looked into the possible causes but couldn't find any:
php max_execution_time is set to 240 seconds and the script never
takes more than one and a half minutes;
sleep documentation says it is per session but curl isn't using cookies so it should be different sessions in every instance (and also if it was using the same it would always block since we always execute three script instances, which it doesn't);
the hosting tech team says there are no errors neither in the server
error log nor in php error log in the timestamp where these issues
happen.
I can't reproduce the issues at will, but they happen at least once a day.
What I'd like to know is what can be interfering with the sleep behaviour? How can I detect or fix it?
Additional information:
linux system
mysql 5.5
apache
php 5.3
php max_execution_time set to 240
Edit 1: Just to clarify: actually we have 3 options, so it writes to 3 log files, one for each option. At any given time there can be up to two instances per option running (each instance of the same option overlaps a small amount of time).
Edit2: As per #Jan suggestion, I added log to the sleep function result. The script stopped once with that log already:
[2016-01-05, 13:11:01] Will sleep at 2016-01-05, 13:11:29
[2016-01-05, 13:11:01] Woke up at 2016-01-05, 13:11:37 with sleep return 5
[2016-01-05, 13:11:01] Not expired at 2016-01-05, 13:11:37
[2016-01-05, 13:11:01] Will sleep at 2016-01-05, 13:11:37
[2016-01-05, 13:11:01] Woke up at 2016-01-05, 13:11:38 with sleep return 13
... no more log from instance [2016-01-05, 13:11:01] ...
[2016-01-05, 13:12:01] Start
According to the sleep documentation:
If the call was interrupted by a signal, sleep() returns a non-zero value. On Windows, this value will always be 192 (the value of the WAIT_IO_COMPLETION constant within the Windows API). On other platforms, the return value will be the number of seconds left to sleep.
So according to the documentation and the log it seems that the sleep is being cut short due to an interrupt.
How can I know what interrupt caused this (pcntl_signal?), where did it come from and is there any way to avoid it?
Edit3: I've added code to handle signals with pcntl_signal (trying to register from signal 1 till 255) and log them, the issue still happens but the log is empty still.
You can define signal handlers with pcntl_signal.
With those handlers you can log when a interruption happens. But AFAIK you can't detect where it comes from.
Also you can use pcntl_alarm for your delayed jobs.
Check PHP Manual - PCNTL Alarm
Given your stack, the interrupt signal could come from apache. Depending on how Apache is communicating with PHP on your stack, there are several timeout options in Apache configuration that could infer with the script execution.
If your cron calls a curl on localhost, maybe it could directly call the PHP file instead, and so avoid using apache processes to keep the request alive? You'll have to edit your dedicated CLI php.ini file with the max_execution_time value.
I have a site hosted on rackspace cloud sites. I have a troubleshooting script I am trying to run to figure out some problems with the site.
Cloud sites has a 30 second timeout and it is timing out before the results page can load. I spoke with their support and they advised me to put a page loading script at the top of the php file to keep the connection open but I have no idea how to do that and the googling I have done hasnt been much help.
The script I am trying to run is too long to include here but if anyone needs it you can find it here http://forum.joomla.org/viewtopic.php?f=621&t=582860
edit: so no matter what I set the execution time to in the script the load balancers rackspace uses will still timeout after 30 seconds. they have told me to run a 'page loading' script at the beginning of the script to keep the connection open so I am about to start looking into how to do that.
You could try the set_time_limit() function:
http://php.net/manual/en/function.set-time-limit.php
By default, a PHP script times out after 30 seconds.
Use the set_time_limit( int $seconds ) function to extend the maximum execution time.
You can also use ini_set() and set the max_execution_time:
ini_set("max_execution_time", 300);
EDIT
if the above doesn't work, then they probably use a secondary mechanism to timeout blocking connections. What you could try in this situation is to flush some data at a regular interval.
ob_start(); // enable output buffering
// output something at regular interval
echo " ";
ob_flush();
// at end of script
ob_end_flush();
Hope this helps.
First of all the script take sometimes to execute and it stops after 30 seconds. Says that the execution time is max about 30 seconds (I know that we can change this param in the httpconf)
but I don't know the max execution time may be 1 hour or more.
So I want to know if there is no time limit execution in php?
Second question: how to show the content in a browser when it's ready and continue the execution, because I always wait the script to end in order to see the content displayed in the browser.
Next time you post a question, make sure to research it better. Both questions are well documented on stackoverflow and the internet.
now to your answer...
To let it run until finished regardless of the time it takes:
http://www.php.net/manual/en/function.set-time-limit.php
set_time_limit(0);
To display content right after it is generated:
http://www.php.net/manual/en/function.flush.php
ob_flush();
flush();
A script that might run that long should be run from the console. Browser timeouts could also interfere with running it via browser.
If you use a console script, you could route the output into a database and query it from there.
You have in php.ini this code. I am not sure you could check it from elsewhere than php.ini.
; Maximum execution time of each script, in seconds
; http://php.net/max-execution-time
; Note: This directive is hardcoded to 0 for the CLI SAPI
max_execution_time = 30
You can't do that in php. The script runs until it finishes execution (normal, or error).
I have a PHP Script, it needs execution time of at least 1000 seconds to complete.
It terminates after around 265 seconds each time with no errors. Since I am using loops I tested number of iterations and it is independent of that, further ruling out a possibility of occurrence of an error in the loop.
I have set max_execution_time to 10800 in php.ini, and also changing memory_limit doesn't affect the results.
Please help! I have scratched my head thoroughly!
Did you check your log file? If you get error 6 or segmentation fault. Then your script is is actually crashing php without showing any errors on the browser (if it is browser and not cli).
If you are using apache on unix, then you should find this log in /var/log/apache2/error.log.
Otherwise you can define the path of the log file in .htaccess by add this line:
php_value error_log "/path/to/somewhere/convenient/php_errors.log"
Change the path to somewhere where your httpd has write permission and where you have read permission.
give:
set_time_limit(0);
in the beginning of script. so that the code execution will not time out unless its completed.
This same error happened to me. One of my PHP functions died without any errors sent to stderr, stdout, nor any other log file.
What happened was I was using a helper PHP script written by some other developer that was setting the memory limit to 512MB half way through the operation of my program. The sub module poisoned the well by also setting the error log settings to silent at some point mid-way through the processing of my script.
You can prove if this is happening to you by printing out the php system settings available to your PHP script on EVERY iteration of the loop. When the subscript did the dirty deed, the PHP engine throws a fit after the garbage collector runs at a random point in the future, then dies immediately without error. It's a bug in the PHP garbage collector when sub modules mess with system settings as the garbage collector is doing its work.
Solution: Edit the php helper sub-modules, and make sure they don't tweak the system settings, as the garbage collector is doing its work. The PHP interpreter will freak out and die without any errors or output at a random interval after the poisoning of the PHP system variables.
On PHP, the only way to hide errors from logs even when the display_errors and error_reporting are properly indicated is to use the Error Control Operators,
for example: the following code will raise a Fatal error but a "hidden" one, it will not appear neither on the logs or on the browser.
#echo "forgetting the last quote;
So set the display_errors value to On and the error_reporting depending on the PHP version you are using (see differences here), and check if you are using a "#" symbol on your code.
I have an upload form that uploads mp3s to my site. I have some intermittent issues with some users which I suspect to be slow upload connections...
But anyway the first line of code is set_time_limit(0); which did fix it for SOME users that had connections that were taking a while to upload, but some are still getting timed out and I have no idea why.
It says the script has exceeded limit execution of 60 seconds. The script has no loops so it's not like it's some kind of infinite loop.
The weird thing is that no matter what line of code is in the first line it will always say "error on line one, two, etc" even if it's set_time_limit(0);. I tried erasing it and the very first line of code always seems to be the error, it doesn't even give me a hint of why it can't execute the php page.
This is an issue only few users are experiencing and no one else seems to be affected. Could anyone throw some ideas as to why this could be happening?
set_time_limt() will only effect the actual execution of the PHP code on the page. You want to set the PHP directive max_input_time, which controls how long the script will accept input (like files) for. The catch is that you need to set this in php.ini, as if the default max_input_time is exceeded, it'll never reach the script which is attempting to change it with ini_set().
Sure, a couple of things noted in the PHP Manual.
Make sure PHP is not running in safe-mode. set_time_limit has no affect when PHP is running in safe_mode.
Second, and this is where I assume your problem lies.....
Note: The set_time_limit() function and the configuration directive max_execution_time only affect the execution time of the script itself. Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real.
So your stream may be the culprit.
Can you post a little of your upload script, are you calling a separate file to handle the upload using Headers?
Try ini_set('max_execution_time', 0); instead.