I am working on a long-running PHP script. The script runs from command line. I need to add a graceful shutdown capability. I tried to use the process control library to handle POSIX signals. Unfortunately, some third party libraries that I am using are also intercepting POSIX signals, and they handle the signal and my script is not informed. I tried using SIGTERM first. Then I tried using custom signals SIGUSR1 and SIGUSR2, same happened.
Is it possible to define/use a new signal that is not defined in POSIX?
If not, what else can I use to implement graceful shutdown?
Related
I'm looking for a way to gracefully exit from a long running python script which I launch from my Laravel app.
My actual process to do so is:
From Laravel set a 'script_state' to 1 in a MySql table database
Launch the python script via shell_exec
The python scripts periodically check the 'script_state' by a MySql query. If it is changed to 0 (intentionally from my Laravel app) then it gracefully exit the script
Retrieving the pid from the shell_exec and then kill could have been an option but I actually wan't the script to stop and exit gracefully.
My actual config works but I'm looking for a better approach.
Retrieving the pid from the shell_exec and then kill could have been an option but I actually wan't the script to stop and exit gracefully.
This is probably your best bet. You can use SIGTERM to politely ask the script to exit:
The SIGTERM signal is a generic signal used to cause program termination. Unlike SIGKILL, this signal can be blocked, handled, and ignored. It is the normal way to politely ask a program to terminate.
This is effectively what happens when you click the close button in a GUI application.
In your Python code you can handle SIGTERM using the signal module with whatever cleanup logic you want, then exit:
import signal, os
def handler(signum, frame):
print('Signal handler called with signal', signum)
raise OSError("Couldn't open device!")
# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)
# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)
signal.alarm(0) # Disable the alarm
See also this Stack Overflow answer for a class that cleans up before exiting.
I am working on an application that has periodically-called background processes. One of these was being called by cron, but I am looking for something more robust, so am converting it to run under Supervisor. (It will probably run for 10 minutes, during which time it can detect work to do, or idle. Once it exits, Supervisor will automatically respawn a clean instance.)
Since Supervisor is better at ensuring that only a specified number of instances of something are running in parallel, I can get away with running them longer. This does mean however that my processes are more likely to receive termination signals, either from kill directly, or because they have been stopped via Supervisor. I am therefore experimenting with how to handle this in PHP.
It looks like the basic solution is to use pcntl_signal() like so:
declare(ticks = 1);
pcntl_signal(SIGTERM, 'signalHandler');
pcntl_signal(SIGINT, 'signalHandler');
function signalHandler($signal) {
switch($signal) {
case SIGTERM:
case SIGINT:
echo "Exiting now...\n";
exit();
}
}
However, I have several points in my code that could do with careful shutdown handling. One approach is to make one handler call those various things, but it would require a bit of refactoring, which I would like to avoid. The alternative is to add pcntl_signal() everywhere I need it, but unfortunately it seems only one handler can be installed at once.
However, it looks like I might be able to use register_shutdown_function(). This does not trap ^C or other termination sigs on its own, and the manual is quite clear on this point:
Shutdown functions will not be executed if the process is killed with a SIGTERM or SIGKILL signal
What is surprising is that I have found that if I employ pcntl_signal() to just do an exit, then the shutdown handlers are indeed called. Furthermore, since one can have many shutdown handlers, this solves my problem nicely - each class in my code that wishes to handle termination gracefully can capture and manage its own shutdown.
My initial question is, why does this work? I have tried registering a shutdown function without a signal handler, and this does not seem to be called, as the manual says. I guess the process is kept alive by PHP in order to handle the signal, which causes shutdown handlers to be called?
Also, can I rely on this behaviour, when the manual casts doubt on it? I am using PHP 5.5, and am not looking to upgrade to PHP7 just yet. So I'd be interested in whether this works on 5.5 and 5.6, on a variety of distributions.
Of course, whether it would (or would not) work on 7.0 and/or 7.1, would be interesting too - I am aware though that ticks will be handled differently in 7.1, so there is a greater chance of this having a different behaviour.
PHP user shutdown functions are called during ordinary process termination, somewhat analogous to functions registered with atexit in other programming languages. An explicit exit or implicit exit at end-of-script is an ordinary process termination.
Signal death, however, is abnormal process termination. The default behavior for SIGTERM (and the mandatory behavior for SIGKILL) is to terminate the running process immediately, without running any cleanup handlers.
When you intercept a SIGTERM with pcntl_signal, you are installing different behavior and giving yourself the opportunity to experience ordinary process termination.
Yes, you can rely on this behavior. While the main manual entry is not explicit, the rest of the "Note" you quote reads:
[Y]ou can use pcntl_signal() to install a handler for a SIGTERM which uses exit() to end cleanly.
In addition to pilcrows correct answer:
The signal handler is necessary for changing the behavior of the signal (i.e. instead of the default action). Whatever happens after that is irrelevant.
You might use the signal handler to just do emergency cleanup and then call posix_kill(getmypid(), SIGKILL); to force termination without cleanup. If you exit() here, normally the shutdown sequence (i.e. calling of all destructors, shutdown handlers etc.) is initiated.
When you say in addition to, you're not strictly correct. You need to use the signal handler instead and may additionally have a shutdown handler. [referring to your comment]
Be also aware that calling exit() after the shutdown sequence has been initiated will abort the current part of it (i.e. when you're currently inside a shutdown handler and exit() gets called, all subsequent shutdown handlers are ignored - but e.g. destructors will still be called). Ideally you have some check against this:
register_shutdown_function(function() { $GLOBALS["#__in_shutdown"] = 1; });
and have a check around exit():
if (empty($GLOBALS["#__in_shutdown"])) {
exit;
}
Just in case you want to be really safe unless someone fires SIGKILL on it.
Note that 7.0 or 7.1 won't change this behavior; all what changes in 7.1 is declare(ticks=1); being superfluous, the exhibited behavior is still the same.
Situation
I have a daemon I wrote in PHP (not the best language for this, but work with me), and it is made to receive jobs from a queue and process them whenever a job needs to be done. For each new job, I use pcntl_fork() to fork the job off into a child process. Within this child process, I then use proc_open() to execute long-running system commands for audio transcoding, which returns directly to the child when finished. When the job is completely done, the child exits and is cleaned up by the parent process.
To keep this daemon always running, I use upstart. Here is my upstart configuration file:
description "Audio Transcoding Daemon"
start on startup
stop on shutdown
# kill signal SIGCHLD
kill timeout 1200 # Don't force kill the process until it runs over 20 minutes
respawn
exec audio-daemon.php
Goal
Because I want to use this daemon in a distributed environment, I want to be able to shutdown the server at any time without disrupting any running jobs. To do this, I have already implemented signal handlers using pcntl_signal() for SIGTERM, SIGHUP, and SIGINT on the parent process, which waits for all children to exit normally before exiting itself. The children also have signal handlers, but they are made to ignore all kill signals.
Problem
The problem is, according to the docs...
The signal specified by the kill signal stanza is sent to the process group of the main process. (such that all processes belonging to the jobs main process are killed). By default this signal is SIGTERM.
This is concerning because, in my child process, I run system commands through proc_open(), which spawns new child processes as well. So, whenever I run sudo stop audio-daemon, this sub-process (which happens to be sox) is killed immediately, and the job returns back with an error. Apparently, sox obeys SIGTERM and does what it's told...
Originally, I thought, "Fine. I'll just change kill signal to send something that is inherently ignored, and I'll just pick it up in the main process only." But according to the manual, there are only two signals that are ignored by default: SIGCHLD and SIGURG (and possibly SIGWINCH). But I'm afraid of getting false flags, since these can also be triggered other ways.
There are ways to create a custom signal using what the manual calls "Real-time Signals" but it also states...
The default action for an unhandled real-time signal is to terminate the receiving process.
So that doesn't help...
Can you think of any way that I can get upstart to keep all of my sub-processes open until they complete? I really don't want to go digging through sox's source code to modify its signal handlers, and while I could set SIGCHLD, SIGURG, or SIGWINCH as my upstart kill signal and pray nothing else sends them my way, I can't help but think there's a better way to do this... Any ideas?
Thanks for all your help! :)
Since I haven't received any other answers for how to do this a better way, this is what I ended up doing, and I hope it helps someone out there...
To stall shutdown/reboot of the system until the daemon is finished, I changed my start on and stop on in my upstart configuration. And to keep upstart from killing my children, I resorted to using SIGURG as my kill signal, which I then catch as a kill signal in my main daemon process only.
Here is my final upstart configuration:
description "Audio Transcoding Daemon"
start on runlevel [2345]
stop on starting rc RUNLEVEL=[016] # Block shutdown/reboot until the daemon ends
kill signal SIGURG # Kill the process group with SIGURG instead of SIGTERM so only the main process will pick it up (since SIGURG will be ignored by all children by default)
kill timeout 1200 # Don't force kill the process until it runs over 20 minutes
respawn
exec audio-daemon.php
Note that using stop on starting rc RUNLEVEL=[016] is necessary to stall shutdown/reboot. stop on runlevel [016] will not work.
Also note that if you use SIGURG in your application for any other reason, using it as a kill signal may cause problems. In my case, I wasn't, so this works fine as far as I can tell.
Ideally, it would be nice if the POSIX standard provided a user-defined signal like SIGUSR1 and SIGUSR2 that was ignored by default. But right now, it looks like it doesn't exist.
Feel free to chime in if you have a better answer, but for now, I hope this helps anyone else having this problem.
Disclaimer: I don't know any PHP
I solved a similar problem with my ruby process by setting a new group id for a launched subprocess. It looks like php has a similar facility.
you can start a new group (detaching from your audio-daemon.php) by settings it's group id to its process id
something like
$chldPid=pcntl_fork()
... << error checks etc
if ($chldPid){
...
posix_setpgid($chldPid, $chldPid)
We are running a PHP Daemon which look into a queue, receives worker jobs and spawns the worker to handle it. The workers themselves acquire a lock on a specific location before proceeding.
We spawn the Daemon as nohup background processes.
This entire architecture seems to work, except when we have to kill the processes, for whatever reason. If we kill them using -9, there is no way to trap it in the worker process and release the locks before dying.
If we use anything less than -9 (like TERM or HUP), it doesn't seem to be received by either the daemon or the worker processes.
Has anybody solved this problem in a better way?
(ps: BTW, Due to other considerations, we may not be able to change our language of implementation, so please only consider PHP based solutions)
I had related problems once too. Let me explain. I had a php 'daemon' that worked like a downloader. It accessed feeds periodically and downloads (laaaarge) content from the net. The daemon had to be stopped at a certain time, lets say 0500 in the morning to prevent it from using the whole bandwith during daytime. I decided to use a cronjob to send SIGTERM to the daemon at 0500.
In the daemon I had the following code:
pcntl_signal(SIGTERM, array($this, 'signal_handler'));
where signal_handler looked like this:
public function signal_handler($signal) {
// some cleanup code
exit(1);
}
Unfortunately this did not work :|
It took me a time to find out what's going on. The first thing I figured out was that I'll have to call the method pcntl_signal_dispatch() on init to enable signal dispatching at all. Quote from the doc (comments):
If you are running PHP as CLI and as a "daemon" (i.e. in a loop), this function must be called in each loop to check if new signals are waiting dispatching.
Ok, so far, it seemed working. But I realized quickly that under certain conditions even this will not work as expected. Sometimes the daemon could only being stopped by kill -9 - as before. :|
So what's the problem?.. Answer: My program called wget to download the files via shell_exec. The problem is, that shell_exec() blocking waits until the child process has terminated. During this blocking wait no signal processing is done, the process can only being terminated using SIGKILL - what is hard. Also a problem was that child processes had to be terminated one by one as they became zombie processes after killing the father.
My solution to this was to execute the child process using proc_open() and the use stream_select() on it's output for non blocking IO.
Now it works like a charm. :) If you need further information don't hesitate to drop a comment.
Note If you are working with PHP < 5.3 then you'll have to use `
declare(ticks=1);
instead of pcntl_signal_dispatch(). You can rfer to the the documentation of pcntl_signal() for that. But if possible you should upgrade to PHP >= 5.3
The problem was solved just by adding ticks:
// tick use required as of PHP 4.3.0
declare(ticks = 1);
Leaving this alone was causing my code not to work.
*(It's unfortunate that the documentation of pcntl_signal doesn't mention it in a lot more attention grabbing way.)*
You need to catch the signal (SIGTERM). This can be achieved via the function pcntl_signal. This will give you the option to perform any necessary functions before calling exit.
Question,
How can I spurn another process within a daemon?
I want to use the pear system daemon library to spurn a daemon and then spurn off processes within that daemon.
So daemon runs
and then a new process is spurn off and does calculation separately
then other processes are spurn off that runs separate from the daemon.
meanwhile, daemon keeps executing code and spurns off more processes
how can I accomplish this?
System_Daemon only handles startup/shutdown handling, general signal handling and logging.
If you want to spawn new processes from your PHP code, you need to use PHP's pcntl functions.
Spurn? I assume you mean spawn.
PHP has lots of functions for creating processes - however (AFAIK) they are all blocking (except for pcntl_exec which replaces the current process)
A quick sift through the documentation for the Pear System Daemon, this only handles the process of daemonizing the process - not of running a server process and handling multiple clients. How you go about implementing this will have a big impact on how you handle starting up new processes.
One solution would be to fork an instance of the current process to handle an incoming connection - there's an example on the socket_accept() doc page. Then it doesn't matter if the process you start is via a blocking call or not.
But a much simlper solution would be not to bother with a daemon / forking / sockets and just invoke it via [x]inetd using stdio
C.
I had the same problem before. The solution I did was to have one system_daemon calling another system_daemon through exec. You need to change the appPidLocation option to run a new instance of the same code.
To see the list of options I looked at the code of system_daemon.