Process respawn and signal handling in PHP - php

Specifics
I have an issue in PHP, when respawned processes are not handling signals, while before respawning, handling working correctly. I narrowed down my code to the very basic:
declare(ticks=1);
register_shutdown_function(function() {
if ($noRethrow = ob_get_contents()) {
ob_end_clean();
exit;
}
system('/usr/bin/nohup /usr/bin/php '.__FILE__. ' 1>/dev/null 2>/dev/null &');
});
function handler($signal)
{
switch ($signal) {
case SIGTERM:
file_put_contents(__FILE__.'.log', sprintf('Terminated [ppid=%s] [pid=%s]'.PHP_EOL, posix_getppid(), posix_getpid()), FILE_APPEND);
ob_start();
echo($signal);
exit;
case SIGCONT:
file_put_contents(__FILE__.'.log', sprintf('Restarted [ppid=%s] [pid=%s]'.PHP_EOL, posix_getppid(), posix_getpid()), FILE_APPEND);
exit;
}
}
pcntl_signal(SIGTERM, 'handler');
pcntl_signal(SIGCONT, 'handler');
while(1) {
if (time() % 5 == 0) {
file_put_contents(__FILE__.'.log', sprintf('Idle [ppid=%s] [pid=%s]'.PHP_EOL, posix_getppid(), posix_getpid()), FILE_APPEND);
}
sleep(1);
}
As you can see, it does following:
Registering shutdown function, in which respawn a process with nohup (so, to ignore SIGHUP when parent process dies)
Registering handler via pcntl_signal() for SIGTERM and SIGCONT. First will just log a message that process was terminated, while second will lead to respawn of the process. It is achieved with ob_* functions, so to pass a flag, what should be done in shutdown function - either exit or respawn.
Logging some information that script is "alive" to log file.
What is happening
So, I'm starting script with:
/usr/bin/nohup /usr/bin/php script.php 1>/dev/null 2>/dev/null &
Then, in log file, there are entries like:
Idle [ppid=7171] [pid=8849]
Idle [ppid=7171] [pid=8849]
Let's say, then I do kill 8849:
Terminated [ppid=7171] [pid=8849]
Thus, it is successful handling of SIGTERM (and script indeed exits). Now, if I instead do kill -18 8849, then I see (18 is numeric value for SIGCONT):
Idle [ppid=7171] [pid=8849]
Restarted [ppid=7171] [pid=8849]
Idle [ppid=1] [pid=8875]
Idle [ppid=1] [pid=8875]
And, therefore: first, SIGCONT was also handled correctly, and, judging by next "Idle" messages, newly spawned instance of script is working well.
Update #1 : I was thinking about stuff with ppid=1 (thus, init global process) and orphan processes signal handling, but it's not the case. Here is log part, which shows that orphan (ppid=1) process isn't the reason: when worker is started by controlling app, it also invokes it with system() command - same way like worker respawns itself. But, after controlling app invokes worker, it has ppid=1 and responds to signals correctly, while if worker respawns itself, new copy is not responding to them, except SIGKILL. So, issue appears only when worker respawns itself.
Update #2 : I tried to analyze what is happening with strace. Now, here are two blocks.
When worker was yet not respawned - strace output. Take a look on lines 4 and 5, this is when I send SIGCONT, thus kill -18 to a process. And then it triggers all the chain: writing to the file, system() call and exiting current process.
When worker was already respawned by itself - strace output. Here, take a look to lines 8 and 9 - they appeared after receiving SIGCONT. First of: looks like process is still somehow receiving a signal, and, second, it ignores the signal. No actions were done, but process was notified by the system that SIGCONT was sent. Why then the process ignores it - is the question (because, if installing of user handler for SIGCONT failed, then it should end execution, while process is not ended). As for SIGKILL, then output for already respawned worker is like:
nanosleep({1, 0}, <unfinished ...>
+++ killed by SIGKILL +++
Which indicates, that signal was received and did what it should do.
The problem
As the process is respawn, it is not reacting neither to SIGTERM, nor to SIGCONT. However, it is still possible to end it with SIGKILL (so, kill -9 PID indeed ends the process). For example, for process above both kill 8875 and kill -18 8875 will do nothing (process will ignore signals and continue to log messages).
However, I would not say that registering signals is failing completely - because it redefines at least SIGTERM (which normally leads to termination, while in this case it is ignored). Also I suspect that ppid = 1 points to some wrong thing, but I can not say for sure now.
Also, I tried any other kind of signals (in fact, it didn't matter what is the signal code, result was always the same)
The question
What could be the reason of such behavior? Is the way, which I'm respawning a process, correct? If not, what are other options which will allow newly spawned process to use user-defined signal handlers correctly?

Solution : Eventually, strace helped to understand the problem. This is as follows:
nanosleep({1, 0}, {0, 294396497}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
restart_syscall(<... resuming interrupted call ...>) = 0
Thus, it shows the signal was received, but ignored. To fully answer the question, I will need to figure out, why process added signals to ignore list, but unblocking them forcefully with pcntl_sigprocmask() is doing the thing:
pcntl_sigprocmask(SIG_UNBLOCK, [SIGTERM, SIGCONT]);
then all goes well and respawned process receives/handles signals as it is intended. I tried to add only SIGCONT for unblocking, for example - and then it was handled correctly, while SIGTERM was blocked, which points to the thing, that it is exactly the reason of failing to dispatch signals.
Resolution : for some reason, when process spawns itself with signal handlers installed, new instance has those signals masked for ignoring. Unmasking them forcefully solves the issue, but why signals are masked in new instance - that is an open question for now.

It's due to the fact, that you spawn a child process by executing system(foo), and then proceeding with dying of the current process. Hence, the process becomes an orphan, and its parent becomes PID 1 (init).
You can see the change using pstree command.
Before:
init─┬─cron
(...)
└─screen─┬─zsh───pstree
├─3*[zsh]
├─zsh───php
└─zsh───vim
After:
init─┬─cron
(...)
└─php
What wikipedia states:
Orphan processes is kind of the opposite situation of zombie processes, since it refers to the case where a parent process terminates before its child processes, in which case these children are said to become "orphaned".
Unlike the asynchronous child-to-parent notification that happens when a child process terminates (via the SIGCHLD signal), child processes are not notified immediately when their parent finishes. Instead, the system simply redefines the "parent-pid" field in the child process's data to be the process that is the "ancestor" of every other process in the system, whose pid generally has the value 1 (one), and whose name is traditionally "init". It is thus said that "init 'adopts' every orphan process on the system".
For your situation, I would suggest two options:
Use two scripts: one for managing the child, and second one, "worker", to actually perform the job,
or, use one script, that will include both: outer part will manage, inner part, forked from outer, will do the job.

Related

MySQL code causes PHP script to crash at popen/exec

I have the following PHP 5.6.19 code on a Ubuntu 14.04 server. This code simply connects to a MySQL 5.6.28 database, waits a minute, launches another process of itself, then exits.
Note: this is the full script, and it's purpose is to demonstrate the problem - it doesn't do anything useful.
class DatabaseConnector {
const DB_HOST = 'localhost';
const DB_NAME = 'database1';
const DB_USERNAME = 'root';
const DB_PASSWORD = 'password';
public static $db;
public static function Init() {
if (DatabaseConnector::$db === null) {
DatabaseConnector::$db = new PDO('mysql:host=' . DatabaseConnector::DB_HOST . ';dbname=' . DatabaseConnector::DB_NAME . ';charset=utf8', DatabaseConnector::DB_USERNAME, DatabaseConnector::DB_PASSWORD);
}
}
}
$startTime = time();
// ***** Script works fine if this line is removed.
DatabaseConnector::Init();
while (true) {
// Sleep for 100 ms.
usleep(100000);
if (time() - $startTime > 60) {
$filePath = __FILE__;
$cmd = "nohup php $filePath > /tmp/1.log 2>&1 &";
// ***** Script sometimes exits here without opening the process and without errors.
$p = popen($cmd, 'r');
pclose($p);
exit;
}
}
I start the first process of the script using nohup php myscript.php > /tmp/1.log 2>&1 &.
This process loop should go on forever but... based on multiple tests, within a day (but not instantly), the process on the server "disappears" without reason. I discovered that the MySQL code is causing the popen code to fail (the script exits without any error or output).
What is happening here?
Notes
The server runs 24/7.
Memory is not an issue.
The database connects correctly.
The file path does not contain spaces.
The same problem exists when using shell_exec or exec instead of popen (and pclose).
I also know that popen is the line that fails because I did further debugging (not shown above) by logging to a file at certain points in the script.
Is the parent process definitely exiting after forking? I had thought pclose would wait for the child to exit before returning.
If it isn't exiting, I'd speculate that because the mySQL connection is never closed, you're eventually hitting its connection limit (or some other limit) as you spawn the tree of child processes.
Edit 1
I've just tried to replicate this. I altered your script to fork every half-second, rather than every minute, and was able to kill it off within about 10 minutes.
It looks like the the repeat creation of child processes is generating ever more FDs, until eventually it can't have any more:
$ lsof | grep type=STREAM | wc -l
240
$ lsof | grep type=STREAM | wc -l
242
...
$ lsof | grep type=STREAM | wc -l
425
$ lsof | grep type=STREAM | wc -l
428
...
And that's because the child's inheriting the parent's FDs (in this case for the mySQL connection) when it forks.
If you close the mySQL connection before popen with (in your case):
DatabaseConnector::$db = null;
The problem will hopefully go away.
I had a similar situation using pcntl_fork() and a MySQL connection. The cause here is probably the same.
Background info
popen() creates a child process. The call to pclose() closes the communication channel and the child process continues to run until it exits. This is when the things start to go out of control.
When a child process completes, the parent process receives a SIGCHLD signal. The parent process here is the PHP interpreter that runs the code you posted. The child process is the one launched using popen() (it doesn't matter what command it runs).
There is a small thing here you probably don't know or you have found in the documentation and ignored it because it doesn't make much sense when one programs in PHP. It is mentioned in the documentation of sleep():
If the call was interrupted by a signal, sleep() returns a non-zero value.
The sleep() PHP function is just a wrapper of the sleep() Linux system call (and usleep() PHP function is a wrapper of the usleep() Linux system call.)
What is not told in the PHP documentation is clearly stated in the documentation of the system calls:
sleep() makes the calling thread sleep until seconds seconds have elapsed or a signal arrives which is not ignored.
Back to your code.
There are two places in your code where the PHP interpreter calls the usleep() Linux system function. One of them is clearly visible: your PHP code invokes it. The other one is hidden (see below).
What happens (the visible part)
Starting with the second iteration, if a child process (created using popen() on a previous iteration) happens to exit while the parent program is inside the usleep(100000) call, the PHP interpreter process receives the SIGCHLD signal and its execution resumes before the time being out. The usleep() returns earlier than expected. Because the timeout is short, this effect is not observable by the naked eye. Put 10 seconds instead of 0.1 seconds and you'll notice it.
However, apart from the broken timeout, this doesn't affect the execution of your code in a fatal manner.
Why it crashes (the invisible part)
The second place where an incoming signal hurts your programs execution is hidden deep inside the code of the PHP interpreter. For some protocol reasons, the MySQL client library uses sleep() and/or usleep() in several places. If the interpreter happens to be inside one of these calls when the SIGCHLD arrives, the MySQL client library code is resumed unexpectedly and, many times, it concludes with the erroneous status "MySQL server has gone away (error 2006)".
It's possible that your code ignores (or swallows) the MySQL error status (because it doesn't expect it to happen in that place). Mine didn't and I spent a couple of days of investigation to find out the facts summarized above.
A solution
The solution for the problem is easy (after you know all the internal details exposed above). It is hinted in the documentation quote above: "a signal arrives which is not ignored".
The signals can be masked (ignored) when their arrival is not desired. The PHP PCNTL extension provides the function pcntl_sigprocmask(). It wraps the sigprocmask() Linux system call that sets what signals can be received by the program from now on (in fact, what signals to be blocked).
There are two strategies you can implement, depending of what you need.
If your program needs to communicate with the database and be notified when the child processed complete then you have to wrap all your database calls within a pair of calls to pcntl_sigprocmask() to block then unblock the SIGCHLD signal.
If you doesn't care when the child processes complete then you just call:
pcntl_sigprocmask(SIG_BLOCK, array(SIGCHLD));
before you start creating any child process (before the while()).
It makes your process ignore the termination of the child processes and lets it run its database queries without undesired interruption.
Warning
The default handling of the SIGCHLD signal is to call wait() in order to let the system cleanup after the completed child process. What happens if the signal is not handled (because its delivery is blocked) is explained in the documentation of wait():
A child that terminates, but has not been waited for becomes a "zombie". The kernel maintains a minimal set of information about the zombie process (PID, termination status, resource usage information) in order to allow the parent to later perform a wait to obtain information about the child. As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes. If a parent process terminates, then its "zombie" children (if any) are adopted by init(1), which automatically performs a wait to remove the zombies.
In plain English, if you block the reception of SIGCHLD signal, then you have to call pcntl_wait() in order to cleanup the zombie child processes.
You can add:
pcntl_wait($status, WNOHANG);
somewhere inside the while loop (just before it ends, for example).
the script exits without any error or output
Not surprising when there's no error checking in the code. However if it really is "crashing", then:
if the cause is trapped by the PHP runtime then it will be trying to log an error. Have you tried delibertely creating an error scenario to varify that the reorting/logging is working as you expect?
if the error is not trapped by the PHP runtime, the the OS should be dumping a corefile - have you checked the OS config? Looked for the core file? Analyzed it?
$cmd = "nohup php $filePath > /tmp/1.log 2>&1 &";
This probably doesn't do what you think it does. When you run a process in the background with most versions of nohup, it still retains a relationship with the parent process; the parent cannot be reaped until the child process exits - and a child is always spawning another child before it does.
This is not a valid way to keep your code running in the background / as a daemon. What the right approach is depends on what you are trying to achieve. Is there a specific reason for attempting to renew the process every 60 seconds?
(You never explicitly close the database connection - this is less of an issue as PHP should do this when exit is invoked).
You might want to read this and this
I suggest that process doesn't exit after pclose. In this case every process holds it's own connection to db. After some time connectons limit of MySQL is reached and new connection fails.
To understand what's going on - add some logs before and after strings DatabaseConnector::Init(); and pclose($p);

How to get process PID started by Symfony?

How can I get PID of process started by Symfony?
The code bellow
$process = new \Symfony\Component\Process\Process('vlc');
$process->start();
return $process->getPid();
returns PID 1488. But there is no process (no vlc, no php) in system with same PID.
Edit
Presented code runs in app/console (Symfony\Component\Console\Command\Command)
Process forking
It's not unlikely for processes to spawn off their UI separately and letting the starter process end normally, i.e.
----> vlc (1488) ---> EOL
|
+--> vlc-ui (??) ---> Application
This behaviour is observable by running the application from the command line and checking whether the prompt returns almost immediately.
Hangup signal
Also note that when a parent process exits (your script), the child process may choose to also exit by listening to SIGHUP signals. If you're not doing so already, you could let your script "live" longer by adding a sleep() statement while you investigate.
Another approach that may work in some situations is doing it inversely like a script to grep parse a 'ps -A' containing the desired process (PHP e.g.) and rip off PID from the result. You don't have control on "who's who" on the result set but have control on "who's" actually running.

Executing a program from PHP hangs APACHE

Hello and thank you in advance for your interest.
During the past two weeks I've been struggling with something that is driving me nuts. I have APACHE (2.2.22) and PHP (5.4.3) installed on my Windows box and I'm trying to call a program from a PHP script which calls another program at the same time. Both programs are written in C/C++ and compiled with MINGW32. Regarding to Windows version, I've tested Windows 2003 Server and Windows 7 Professional and both give me the same problems.
Let me introduce these two programs:
1) mytask.exe: this is a program that is to be executed in background and that periodically populates its status to a file.
2) job.exe: this is the program I want to call from the PHP script. Its goal is to spawn mytask.exe as an independent process (not as a thread).
If I run from a Console window the command below, then job.exe immediately returns and leaves mytask.exe running on the background until it terminates.
> job.exe spawn mytask.exe
jobid=18874111458879FED
Note that job.exe dumps an identifier which is used to manage mytask.exe. For example:
> job.exe status 18874111458879FED
RUNNING
I've checked that if I run the first command from a PHP script, the PHP script randomly blocks forever. If I look to the Windows's task manager, I can see that job.exe is there in a zombie-like state. I can assert that job.exe effectively reaches the usual return 0; statement in its main() routine, so it seems to be something under the wood, in the C runtime.
Furthermore, if I write a simple mytask.exe that simply sleeps for 10 seconds then the PHP script blocks for 10 seconds too (or blocks forever following the random behavior I've just mentioned). In other words, I have no way to make job.exe spawning a process without waiting for it to end, when I call job.exe from a PHP script.
So: there's something I'm doing wrong when spawning mytask.exe and, now, here comes the second part of this digression.
I use the WINAPI function CreateProcess() to spawn the tasks from job.exe. As of the MSDN documentation, I call CreateProcess with bInheritHandles = FALSE, to avoid the child process to yield I/O deadlocks with the PHP script. I also close the process handles returned by CreateProcess() in the PROCESS_INFORMATION structure. The only thing I don't do is waiting for the process to end. On the other hand, regarding to the PHP side, I've tried both exec() and proc_open() PHP functions to call job.exe with no success.
My last observations, though, seem to be in the right way, yet they do not convince me because I don't understand why they work somehow. The fact is that if mytask.exe does fclose(stdout) before sleeping, then the PHP script returns immediately. BUT, HOW??? I told CreateProcess() to not inherit handles, so why am I getting these results? Anyway, I cannot stick with this patch because programs launched by job.exe may not know about who is calling them, so closing stdout from those programs is not a good solution. In UNIX, things are so simple... One just calls fork(), closes standard streams and then calls execve to call the program. In Windows, I've also tried to create a wrapper thread with CreateThread() (to emulate fork()) and then call CreateProcess() from that thread after closing standard streams... but that closed the streams of job.exe too!
All this question could be synthesized in a single one: how can I execute from PHP a program that creates other processes?
I hope somebody could shed some light on this issue... Thank you very much!
I think I've nailed the solution, which is divided in two parts:
1) Regarding the fact the main process stops until the child process ends.
As of MSDN documentation, this is the definition of CreateProcess():
BOOL WINAPI CreateProcess(
_In_opt_ LPCTSTR lpApplicationName,
_Inout_opt_ LPTSTR lpCommandLine,
_In_opt_ LPSECURITY_ATTRIBUTES lpProcessAttributes,
_In_opt_ LPSECURITY_ATTRIBUTES lpThreadAttributes,
_In_ BOOL bInheritHandles,
_In_ DWORD dwCreationFlags,
_In_opt_ LPVOID lpEnvironment,
_In_opt_ LPCTSTR lpCurrentDirectory,
_In_ LPSTARTUPINFO lpStartupInfo,
_Out_ LPPROCESS_INFORMATION lpProcessInformation
);
As I said in my question, I pass FALSE to bInheritHandles, but I was also passing 0 to dwCreationFlags. After a little bit of more research, I found that there's a flag called DETACHED_PROCESS, for which MSDN says:
For console processes, the new process does not inherit its parent's console (the default). The new process can call the AllocConsole function at a later time to create a console. For more information, see Creation of a Console.
Now, job.exe returns immediately despite the fact the child process continues its execution.
2) Regarding the fact the PHP script randomly hangs when calling exec()
It seems to be a bug of PHP. Running exec() family functions in the context of a PHP session may make APACHE to randomly hang, being necessary to restart the server. I found a thread in the Internet in which a user noticed that closing the session (thru session_write_close()) before calling exec() would prevent the script from hanging. The same applies for the proc_open/proc_close functions. So, my script now looks like this:
session_write_close(); //Close the session before proc_open()
$proc = proc_open($cmd,$pipedesc,$pipes);
//do stuff with pipes...
//... and close pipes
$retval = proc_close($proc);
session_start(); //restore session
Hope this helps.

most simple way to start a new process/thread in PHP

Scenario:
Shared hosting, so no ability to install new extensions + no CRON
A submitted request needs to perform some heavy processes.
I want the answer to the client to go as fast as possible, and the heavy lifting to continue immediately, but not stop the client.
can be on a new thread (if it is possible) also no problem with starting a new process.
What is the best way to do this?
On *nix:
exec('/path/to/executable > /dev/null 2>&1 &');
On Windows:
$WshShell = new COM('WScript.Shell');
$oExec = $WshShell->Run('C:\path\to\executable.exe', 0, false);
Both of these will spawn a new process that will run a-synchronously, completely disconnected from the parent. As long as your host allows you to do it.
You can google by key: php continue processing after closing connection.
The following links that relate to your problem, are:
Continue processing after closing connection
http://php-fpm.org/wiki/Features#fastcgi_finish_request.28.29
http://www.php.net/manual/en/features.connection-handling.php
You can use belong command to continue executing without user aborting
ignore_user_abort(true);
set_time_limit(0);
You use fastcgi_finish_request to alert client to stop the response output. And your scripts will continue to be executed.
An example:
// redirecting...
ignore_user_abort(true);
set_time_limit(0);
header("Location: ".$redirectUrl, true);
header("Connection: close", true);
header("Content-Length: 0", true);
ob_end_flush();
flush();
fastcgi_finish_request(); // important when using php-fpm!
sleep (5); // User won't feel this sleep because he'll already be away
// do some work after user has been redirected
To complement #DaveRandom's answer: you don't actually need to redirect STDERR to STDOUT (with 2>&1).
You need to redirect STDOUT though, if you want to prevent the parent process from hanging waiting for the child to finish. This is required cause exec will return the last line of the output of the child process, therefore, it needs to wait for the child's STDOUT to be closed.
That doesn't mean you need to redirect it to /dev/null. You can redirect it to some other file, or even to some other file descriptor (like STDERR: 1>&2).
exec('/path/to/executable'): will start a new process and wait for it to finish (i.e. blocking the parent process).
exec('/path/to/executable &'): basically the same as the above.
$pid = exec('/path/to/executable > /dev/null & echo $!'): will start a process in the background, with child and parent processes running in parallel. The output of /path/to/executable will be discarded, and $pid will receive the PID of the child process.
Using 2>&1 is actually not necessary, cause exec ignores the STDERR of the child process. It is also probably undesirable cause it will make it harder to find some errors, cause they will be just silently thrown away. If you omit 2>&1, you can pipe the STDERR of the parent process to some log file, that can be checked later when something goes wrong:
php /path/to/script.php 2>> /var/log/script_error.log
By using the above to start the script which triggers the child processes, everything that script.php and any child process write to STDERR will be written to the log file.
There are no threads in PHP. You could cheat by sending back an HTML page that triggers an Ajax call to start the heavy process in a new request. But if it's shared hosting, my guess is that you'll quickly hit the limits on memory, time or CPU usage imposed by your hosting provider.
$WshShell = new COM('WScript.Shell');
$oExec = $WshShell->Run('C:\xampp\php\php.exe C:\xampp\htdocs\test.php -a asdf', 0, true);
Cannot pass argv to test.php.
var_dump($argv);

getting tty of caller in signal handler in php

I just wrote php daemon for my app and want to implement some output information generated on specified signal (SIGUSR1). I got signal handler in code
pcntl_signal(SIGUSR1, array($this, 'signal_status'));
and function handler prepared
protected function signal_status($signal_number)
{ printf("blah\n"); }
Everything work except one thing. When i send signal to my daemon (posix_kill($pid, SIGUSR1) or even simple kill -10 pid in bash) i got output in console that starts daemon.
Is it possible to get file descriptor of caller and not of the daemon? I wan't to send this data to specified output (for example after kill -10 PID) and not into FD of daemon.
I hope i wrote this clearly :)
Well, you can not, sending a signal is just setting up some operating system primitives, it has nothing to do with setting up a communication path between your daemon on one hand and the tool used to generate the signal on the other hand. The alternatives you have are either watching the console output of the daemon, or making the daemon dump status to a logfile and create some sort of utility to send the signal and print the logfile (and if you're going that way, why not throw out the logfile altogether and setup a periodical dump of your logging anyhow, since signals are not a polite way of doing inteprocess communiation.

Categories