Related
I have the following PHP 5.6.19 code on a Ubuntu 14.04 server. This code simply connects to a MySQL 5.6.28 database, waits a minute, launches another process of itself, then exits.
Note: this is the full script, and it's purpose is to demonstrate the problem - it doesn't do anything useful.
class DatabaseConnector {
const DB_HOST = 'localhost';
const DB_NAME = 'database1';
const DB_USERNAME = 'root';
const DB_PASSWORD = 'password';
public static $db;
public static function Init() {
if (DatabaseConnector::$db === null) {
DatabaseConnector::$db = new PDO('mysql:host=' . DatabaseConnector::DB_HOST . ';dbname=' . DatabaseConnector::DB_NAME . ';charset=utf8', DatabaseConnector::DB_USERNAME, DatabaseConnector::DB_PASSWORD);
}
}
}
$startTime = time();
// ***** Script works fine if this line is removed.
DatabaseConnector::Init();
while (true) {
// Sleep for 100 ms.
usleep(100000);
if (time() - $startTime > 60) {
$filePath = __FILE__;
$cmd = "nohup php $filePath > /tmp/1.log 2>&1 &";
// ***** Script sometimes exits here without opening the process and without errors.
$p = popen($cmd, 'r');
pclose($p);
exit;
}
}
I start the first process of the script using nohup php myscript.php > /tmp/1.log 2>&1 &.
This process loop should go on forever but... based on multiple tests, within a day (but not instantly), the process on the server "disappears" without reason. I discovered that the MySQL code is causing the popen code to fail (the script exits without any error or output).
What is happening here?
Notes
The server runs 24/7.
Memory is not an issue.
The database connects correctly.
The file path does not contain spaces.
The same problem exists when using shell_exec or exec instead of popen (and pclose).
I also know that popen is the line that fails because I did further debugging (not shown above) by logging to a file at certain points in the script.
Is the parent process definitely exiting after forking? I had thought pclose would wait for the child to exit before returning.
If it isn't exiting, I'd speculate that because the mySQL connection is never closed, you're eventually hitting its connection limit (or some other limit) as you spawn the tree of child processes.
Edit 1
I've just tried to replicate this. I altered your script to fork every half-second, rather than every minute, and was able to kill it off within about 10 minutes.
It looks like the the repeat creation of child processes is generating ever more FDs, until eventually it can't have any more:
$ lsof | grep type=STREAM | wc -l
240
$ lsof | grep type=STREAM | wc -l
242
...
$ lsof | grep type=STREAM | wc -l
425
$ lsof | grep type=STREAM | wc -l
428
...
And that's because the child's inheriting the parent's FDs (in this case for the mySQL connection) when it forks.
If you close the mySQL connection before popen with (in your case):
DatabaseConnector::$db = null;
The problem will hopefully go away.
I had a similar situation using pcntl_fork() and a MySQL connection. The cause here is probably the same.
Background info
popen() creates a child process. The call to pclose() closes the communication channel and the child process continues to run until it exits. This is when the things start to go out of control.
When a child process completes, the parent process receives a SIGCHLD signal. The parent process here is the PHP interpreter that runs the code you posted. The child process is the one launched using popen() (it doesn't matter what command it runs).
There is a small thing here you probably don't know or you have found in the documentation and ignored it because it doesn't make much sense when one programs in PHP. It is mentioned in the documentation of sleep():
If the call was interrupted by a signal, sleep() returns a non-zero value.
The sleep() PHP function is just a wrapper of the sleep() Linux system call (and usleep() PHP function is a wrapper of the usleep() Linux system call.)
What is not told in the PHP documentation is clearly stated in the documentation of the system calls:
sleep() makes the calling thread sleep until seconds seconds have elapsed or a signal arrives which is not ignored.
Back to your code.
There are two places in your code where the PHP interpreter calls the usleep() Linux system function. One of them is clearly visible: your PHP code invokes it. The other one is hidden (see below).
What happens (the visible part)
Starting with the second iteration, if a child process (created using popen() on a previous iteration) happens to exit while the parent program is inside the usleep(100000) call, the PHP interpreter process receives the SIGCHLD signal and its execution resumes before the time being out. The usleep() returns earlier than expected. Because the timeout is short, this effect is not observable by the naked eye. Put 10 seconds instead of 0.1 seconds and you'll notice it.
However, apart from the broken timeout, this doesn't affect the execution of your code in a fatal manner.
Why it crashes (the invisible part)
The second place where an incoming signal hurts your programs execution is hidden deep inside the code of the PHP interpreter. For some protocol reasons, the MySQL client library uses sleep() and/or usleep() in several places. If the interpreter happens to be inside one of these calls when the SIGCHLD arrives, the MySQL client library code is resumed unexpectedly and, many times, it concludes with the erroneous status "MySQL server has gone away (error 2006)".
It's possible that your code ignores (or swallows) the MySQL error status (because it doesn't expect it to happen in that place). Mine didn't and I spent a couple of days of investigation to find out the facts summarized above.
A solution
The solution for the problem is easy (after you know all the internal details exposed above). It is hinted in the documentation quote above: "a signal arrives which is not ignored".
The signals can be masked (ignored) when their arrival is not desired. The PHP PCNTL extension provides the function pcntl_sigprocmask(). It wraps the sigprocmask() Linux system call that sets what signals can be received by the program from now on (in fact, what signals to be blocked).
There are two strategies you can implement, depending of what you need.
If your program needs to communicate with the database and be notified when the child processed complete then you have to wrap all your database calls within a pair of calls to pcntl_sigprocmask() to block then unblock the SIGCHLD signal.
If you doesn't care when the child processes complete then you just call:
pcntl_sigprocmask(SIG_BLOCK, array(SIGCHLD));
before you start creating any child process (before the while()).
It makes your process ignore the termination of the child processes and lets it run its database queries without undesired interruption.
Warning
The default handling of the SIGCHLD signal is to call wait() in order to let the system cleanup after the completed child process. What happens if the signal is not handled (because its delivery is blocked) is explained in the documentation of wait():
A child that terminates, but has not been waited for becomes a "zombie". The kernel maintains a minimal set of information about the zombie process (PID, termination status, resource usage information) in order to allow the parent to later perform a wait to obtain information about the child. As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes. If a parent process terminates, then its "zombie" children (if any) are adopted by init(1), which automatically performs a wait to remove the zombies.
In plain English, if you block the reception of SIGCHLD signal, then you have to call pcntl_wait() in order to cleanup the zombie child processes.
You can add:
pcntl_wait($status, WNOHANG);
somewhere inside the while loop (just before it ends, for example).
the script exits without any error or output
Not surprising when there's no error checking in the code. However if it really is "crashing", then:
if the cause is trapped by the PHP runtime then it will be trying to log an error. Have you tried delibertely creating an error scenario to varify that the reorting/logging is working as you expect?
if the error is not trapped by the PHP runtime, the the OS should be dumping a corefile - have you checked the OS config? Looked for the core file? Analyzed it?
$cmd = "nohup php $filePath > /tmp/1.log 2>&1 &";
This probably doesn't do what you think it does. When you run a process in the background with most versions of nohup, it still retains a relationship with the parent process; the parent cannot be reaped until the child process exits - and a child is always spawning another child before it does.
This is not a valid way to keep your code running in the background / as a daemon. What the right approach is depends on what you are trying to achieve. Is there a specific reason for attempting to renew the process every 60 seconds?
(You never explicitly close the database connection - this is less of an issue as PHP should do this when exit is invoked).
You might want to read this and this
I suggest that process doesn't exit after pclose. In this case every process holds it's own connection to db. After some time connectons limit of MySQL is reached and new connection fails.
To understand what's going on - add some logs before and after strings DatabaseConnector::Init(); and pclose($p);
i have the name of a long running process like ffmpeg (or more) and i want to kill it - or them - in php and without using the exec
Is that possible ? Its just like killall ffmpeg linux command .
You can kill process in safe way using the posix function "posix_kill".
Example:
// We assumes that 1223 PID is one ffmpeg process
posix_kill(1223, SIGHUP); // This send a polite termination signal to the process
posix_kill(1223, SIGINT); // This send a interrupt signal to the process
posix_kill(1223, SIGTERM); // This send the termination signal and tried to clean-up the process (Use it if SIGHUP and SIGINT fails)
posix_kill(1223, SIGKILL); // This send an unpolite signal to the kernel explaining that we want to kill the process ASAP and unconditionally and we don't care if we loss some data and the process is going to generate some memory leak (Use this as last option).
Now the problem is that you want to have the ffmpeg PID's, you can achieve that in a non-intrusive way reading the processes information from the "proc" filesystem.
Example that provides the process name:
/proc/1223/cmdline
In order to get the higher/maximum process number, you have to read the file /proc/sys/kernel/pid_max. Now you have to iterate across the process information in the "proc" filesystems until the pid_max is reached. You have have to check that process directory exists and that string returned from the /proc/xxxx/cmdline is contain the string "ffmpeg" or "libav" (Some linux distributions have renamed ffmpeg by libav or something like that).
Remember that in order to kill the ffmpeg processes the PHP script owner (Maybe Apache?) should be added to the same group that launch the ffmpeg processes.
Actually you can use linux commands via "exec" function, i don't know any other way. Check this out :
http://php.net/manual/en/function.exec.php
Specifics
I have an issue in PHP, when respawned processes are not handling signals, while before respawning, handling working correctly. I narrowed down my code to the very basic:
declare(ticks=1);
register_shutdown_function(function() {
if ($noRethrow = ob_get_contents()) {
ob_end_clean();
exit;
}
system('/usr/bin/nohup /usr/bin/php '.__FILE__. ' 1>/dev/null 2>/dev/null &');
});
function handler($signal)
{
switch ($signal) {
case SIGTERM:
file_put_contents(__FILE__.'.log', sprintf('Terminated [ppid=%s] [pid=%s]'.PHP_EOL, posix_getppid(), posix_getpid()), FILE_APPEND);
ob_start();
echo($signal);
exit;
case SIGCONT:
file_put_contents(__FILE__.'.log', sprintf('Restarted [ppid=%s] [pid=%s]'.PHP_EOL, posix_getppid(), posix_getpid()), FILE_APPEND);
exit;
}
}
pcntl_signal(SIGTERM, 'handler');
pcntl_signal(SIGCONT, 'handler');
while(1) {
if (time() % 5 == 0) {
file_put_contents(__FILE__.'.log', sprintf('Idle [ppid=%s] [pid=%s]'.PHP_EOL, posix_getppid(), posix_getpid()), FILE_APPEND);
}
sleep(1);
}
As you can see, it does following:
Registering shutdown function, in which respawn a process with nohup (so, to ignore SIGHUP when parent process dies)
Registering handler via pcntl_signal() for SIGTERM and SIGCONT. First will just log a message that process was terminated, while second will lead to respawn of the process. It is achieved with ob_* functions, so to pass a flag, what should be done in shutdown function - either exit or respawn.
Logging some information that script is "alive" to log file.
What is happening
So, I'm starting script with:
/usr/bin/nohup /usr/bin/php script.php 1>/dev/null 2>/dev/null &
Then, in log file, there are entries like:
Idle [ppid=7171] [pid=8849]
Idle [ppid=7171] [pid=8849]
Let's say, then I do kill 8849:
Terminated [ppid=7171] [pid=8849]
Thus, it is successful handling of SIGTERM (and script indeed exits). Now, if I instead do kill -18 8849, then I see (18 is numeric value for SIGCONT):
Idle [ppid=7171] [pid=8849]
Restarted [ppid=7171] [pid=8849]
Idle [ppid=1] [pid=8875]
Idle [ppid=1] [pid=8875]
And, therefore: first, SIGCONT was also handled correctly, and, judging by next "Idle" messages, newly spawned instance of script is working well.
Update #1 : I was thinking about stuff with ppid=1 (thus, init global process) and orphan processes signal handling, but it's not the case. Here is log part, which shows that orphan (ppid=1) process isn't the reason: when worker is started by controlling app, it also invokes it with system() command - same way like worker respawns itself. But, after controlling app invokes worker, it has ppid=1 and responds to signals correctly, while if worker respawns itself, new copy is not responding to them, except SIGKILL. So, issue appears only when worker respawns itself.
Update #2 : I tried to analyze what is happening with strace. Now, here are two blocks.
When worker was yet not respawned - strace output. Take a look on lines 4 and 5, this is when I send SIGCONT, thus kill -18 to a process. And then it triggers all the chain: writing to the file, system() call and exiting current process.
When worker was already respawned by itself - strace output. Here, take a look to lines 8 and 9 - they appeared after receiving SIGCONT. First of: looks like process is still somehow receiving a signal, and, second, it ignores the signal. No actions were done, but process was notified by the system that SIGCONT was sent. Why then the process ignores it - is the question (because, if installing of user handler for SIGCONT failed, then it should end execution, while process is not ended). As for SIGKILL, then output for already respawned worker is like:
nanosleep({1, 0}, <unfinished ...>
+++ killed by SIGKILL +++
Which indicates, that signal was received and did what it should do.
The problem
As the process is respawn, it is not reacting neither to SIGTERM, nor to SIGCONT. However, it is still possible to end it with SIGKILL (so, kill -9 PID indeed ends the process). For example, for process above both kill 8875 and kill -18 8875 will do nothing (process will ignore signals and continue to log messages).
However, I would not say that registering signals is failing completely - because it redefines at least SIGTERM (which normally leads to termination, while in this case it is ignored). Also I suspect that ppid = 1 points to some wrong thing, but I can not say for sure now.
Also, I tried any other kind of signals (in fact, it didn't matter what is the signal code, result was always the same)
The question
What could be the reason of such behavior? Is the way, which I'm respawning a process, correct? If not, what are other options which will allow newly spawned process to use user-defined signal handlers correctly?
Solution : Eventually, strace helped to understand the problem. This is as follows:
nanosleep({1, 0}, {0, 294396497}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
restart_syscall(<... resuming interrupted call ...>) = 0
Thus, it shows the signal was received, but ignored. To fully answer the question, I will need to figure out, why process added signals to ignore list, but unblocking them forcefully with pcntl_sigprocmask() is doing the thing:
pcntl_sigprocmask(SIG_UNBLOCK, [SIGTERM, SIGCONT]);
then all goes well and respawned process receives/handles signals as it is intended. I tried to add only SIGCONT for unblocking, for example - and then it was handled correctly, while SIGTERM was blocked, which points to the thing, that it is exactly the reason of failing to dispatch signals.
Resolution : for some reason, when process spawns itself with signal handlers installed, new instance has those signals masked for ignoring. Unmasking them forcefully solves the issue, but why signals are masked in new instance - that is an open question for now.
It's due to the fact, that you spawn a child process by executing system(foo), and then proceeding with dying of the current process. Hence, the process becomes an orphan, and its parent becomes PID 1 (init).
You can see the change using pstree command.
Before:
init─┬─cron
(...)
└─screen─┬─zsh───pstree
├─3*[zsh]
├─zsh───php
└─zsh───vim
After:
init─┬─cron
(...)
└─php
What wikipedia states:
Orphan processes is kind of the opposite situation of zombie processes, since it refers to the case where a parent process terminates before its child processes, in which case these children are said to become "orphaned".
Unlike the asynchronous child-to-parent notification that happens when a child process terminates (via the SIGCHLD signal), child processes are not notified immediately when their parent finishes. Instead, the system simply redefines the "parent-pid" field in the child process's data to be the process that is the "ancestor" of every other process in the system, whose pid generally has the value 1 (one), and whose name is traditionally "init". It is thus said that "init 'adopts' every orphan process on the system".
For your situation, I would suggest two options:
Use two scripts: one for managing the child, and second one, "worker", to actually perform the job,
or, use one script, that will include both: outer part will manage, inner part, forked from outer, will do the job.
I'm starting my php script in the following way:
bash
cd 'path'
php -f 'scriptname'.php
There is no output while the php script is running.
After a time, the php script responds with:
Killed
My idea is that it reached the memory_limit: ini_set('memory_limit', '40960M');
Increasing the memory limit seemed to solve the problem, but it only increased the edge.
What exactly does that Killed phrase mean?
Your process is killed. There could be a multitude of reasons, but it's easy to discard some of the more obvious.
php limits: if you run into a php limit, you'll get an error in the logfile, and probably on the commandline as well. This normally does not print 'killed'
the session-is-ended-issues: if you still have your session, then your session is obvioiusly not ended, so disregard all the nohup and & stuff
If your server is starved for resources (no memory, no swap), the kernel might kill your process. This is probably what's happening.
In anycase: your process is getting send a signal that it should stop. Normally only a couple of 'things' can do this
your account (e.g. you kill the process)
an admin user (e.g. root)
the kernel when it is really needing your memory for itself.
maybe some automated process, for instance, if you live on a shared server and you take up more then your share of resources.
references: Who "Killed" my process and why?
You could be running out of memory in the PHP script. Here is how to reproduce that error:
I'm doing this example on Ubuntu 12.10 with PHP 5.3.10:
Create this PHP script called m.php and save it:
<?php
function repeat(){
repeat();
}
repeat();
?>
Run it:
el#apollo:~/foo$ php m.php
Killed
The program takes 100% CPU for about 15 seconds then stops. Look at dmesg | grep php and there are clues:
el#apollo:~/foo$ dmesg | grep php
[2387779.707894] Out of memory: Kill process 2114 (php) score 868 or
sacrifice child
So in my case, the PHP program printed "Killed" and halted because it ran out of memory due to an infinite loop.
Solutions:
Increase the amount of RAM available.
Break down the problem set into smaller chunks that operate sequentially.
Rewrite the program so it has a much smaller memory requirements.
Killed is what bash says when a process exits after a SIGKILL, it's not related to putty.
Terminated is what bash says when a process exits after a a SIGTERM.
You are not running into PHP limits, you may be running into a different problem, see:
Return code when OOM killer kills a process
http://en.wikipedia.org/wiki/Nohup
Try using nohup before your command.
nohup catches the hangup signal while the ampersand doesn't (except the shell is confgured that way or doesn't send SIGHUP at all).
Normally, when running a command using & and exiting the shell afterwards, the shell will terminate the sub-command with the hangup signal (kill -SIGHUP ). This can be prevented using nohup, as it catches the signal and ignores it so that it never reaches the actual application.
In case you're using bash, you can use the command shopt | grep hupon to find out whether your shell sends SIGHUP to its child processes or not. If it is off, processes won't be terminated, as it seems to be the case for you.
There are cases where nohup does not work, for example when the process you start reconnects the NOHUP signal.
nohup php -f 'yourscript'.php
If you are already taking care of php.ini settings related with script memory and timeout then may be its linux ssh connection which terminating in active session or some thing like that.
You can use 'nohup' linux command run a command immune to hangups
shell> nohup php -f 'scriptname'.php
Edit:- You can close your session by adding '&' at end of command:-
shell> nohup php -f 'scriptname'.php &> /dev/null &
'&' operater at end of any comand in linux move that command in background
Hello and thank you in advance for your interest.
During the past two weeks I've been struggling with something that is driving me nuts. I have APACHE (2.2.22) and PHP (5.4.3) installed on my Windows box and I'm trying to call a program from a PHP script which calls another program at the same time. Both programs are written in C/C++ and compiled with MINGW32. Regarding to Windows version, I've tested Windows 2003 Server and Windows 7 Professional and both give me the same problems.
Let me introduce these two programs:
1) mytask.exe: this is a program that is to be executed in background and that periodically populates its status to a file.
2) job.exe: this is the program I want to call from the PHP script. Its goal is to spawn mytask.exe as an independent process (not as a thread).
If I run from a Console window the command below, then job.exe immediately returns and leaves mytask.exe running on the background until it terminates.
> job.exe spawn mytask.exe
jobid=18874111458879FED
Note that job.exe dumps an identifier which is used to manage mytask.exe. For example:
> job.exe status 18874111458879FED
RUNNING
I've checked that if I run the first command from a PHP script, the PHP script randomly blocks forever. If I look to the Windows's task manager, I can see that job.exe is there in a zombie-like state. I can assert that job.exe effectively reaches the usual return 0; statement in its main() routine, so it seems to be something under the wood, in the C runtime.
Furthermore, if I write a simple mytask.exe that simply sleeps for 10 seconds then the PHP script blocks for 10 seconds too (or blocks forever following the random behavior I've just mentioned). In other words, I have no way to make job.exe spawning a process without waiting for it to end, when I call job.exe from a PHP script.
So: there's something I'm doing wrong when spawning mytask.exe and, now, here comes the second part of this digression.
I use the WINAPI function CreateProcess() to spawn the tasks from job.exe. As of the MSDN documentation, I call CreateProcess with bInheritHandles = FALSE, to avoid the child process to yield I/O deadlocks with the PHP script. I also close the process handles returned by CreateProcess() in the PROCESS_INFORMATION structure. The only thing I don't do is waiting for the process to end. On the other hand, regarding to the PHP side, I've tried both exec() and proc_open() PHP functions to call job.exe with no success.
My last observations, though, seem to be in the right way, yet they do not convince me because I don't understand why they work somehow. The fact is that if mytask.exe does fclose(stdout) before sleeping, then the PHP script returns immediately. BUT, HOW??? I told CreateProcess() to not inherit handles, so why am I getting these results? Anyway, I cannot stick with this patch because programs launched by job.exe may not know about who is calling them, so closing stdout from those programs is not a good solution. In UNIX, things are so simple... One just calls fork(), closes standard streams and then calls execve to call the program. In Windows, I've also tried to create a wrapper thread with CreateThread() (to emulate fork()) and then call CreateProcess() from that thread after closing standard streams... but that closed the streams of job.exe too!
All this question could be synthesized in a single one: how can I execute from PHP a program that creates other processes?
I hope somebody could shed some light on this issue... Thank you very much!
I think I've nailed the solution, which is divided in two parts:
1) Regarding the fact the main process stops until the child process ends.
As of MSDN documentation, this is the definition of CreateProcess():
BOOL WINAPI CreateProcess(
_In_opt_ LPCTSTR lpApplicationName,
_Inout_opt_ LPTSTR lpCommandLine,
_In_opt_ LPSECURITY_ATTRIBUTES lpProcessAttributes,
_In_opt_ LPSECURITY_ATTRIBUTES lpThreadAttributes,
_In_ BOOL bInheritHandles,
_In_ DWORD dwCreationFlags,
_In_opt_ LPVOID lpEnvironment,
_In_opt_ LPCTSTR lpCurrentDirectory,
_In_ LPSTARTUPINFO lpStartupInfo,
_Out_ LPPROCESS_INFORMATION lpProcessInformation
);
As I said in my question, I pass FALSE to bInheritHandles, but I was also passing 0 to dwCreationFlags. After a little bit of more research, I found that there's a flag called DETACHED_PROCESS, for which MSDN says:
For console processes, the new process does not inherit its parent's console (the default). The new process can call the AllocConsole function at a later time to create a console. For more information, see Creation of a Console.
Now, job.exe returns immediately despite the fact the child process continues its execution.
2) Regarding the fact the PHP script randomly hangs when calling exec()
It seems to be a bug of PHP. Running exec() family functions in the context of a PHP session may make APACHE to randomly hang, being necessary to restart the server. I found a thread in the Internet in which a user noticed that closing the session (thru session_write_close()) before calling exec() would prevent the script from hanging. The same applies for the proc_open/proc_close functions. So, my script now looks like this:
session_write_close(); //Close the session before proc_open()
$proc = proc_open($cmd,$pipedesc,$pipes);
//do stuff with pipes...
//... and close pipes
$retval = proc_close($proc);
session_start(); //restore session
Hope this helps.