Scenario:
Shared hosting, so no ability to install new extensions + no CRON
A submitted request needs to perform some heavy processes.
I want the answer to the client to go as fast as possible, and the heavy lifting to continue immediately, but not stop the client.
can be on a new thread (if it is possible) also no problem with starting a new process.
What is the best way to do this?
On *nix:
exec('/path/to/executable > /dev/null 2>&1 &');
On Windows:
$WshShell = new COM('WScript.Shell');
$oExec = $WshShell->Run('C:\path\to\executable.exe', 0, false);
Both of these will spawn a new process that will run a-synchronously, completely disconnected from the parent. As long as your host allows you to do it.
You can google by key: php continue processing after closing connection.
The following links that relate to your problem, are:
Continue processing after closing connection
http://php-fpm.org/wiki/Features#fastcgi_finish_request.28.29
http://www.php.net/manual/en/features.connection-handling.php
You can use belong command to continue executing without user aborting
ignore_user_abort(true);
set_time_limit(0);
You use fastcgi_finish_request to alert client to stop the response output. And your scripts will continue to be executed.
An example:
// redirecting...
ignore_user_abort(true);
set_time_limit(0);
header("Location: ".$redirectUrl, true);
header("Connection: close", true);
header("Content-Length: 0", true);
ob_end_flush();
flush();
fastcgi_finish_request(); // important when using php-fpm!
sleep (5); // User won't feel this sleep because he'll already be away
// do some work after user has been redirected
To complement #DaveRandom's answer: you don't actually need to redirect STDERR to STDOUT (with 2>&1).
You need to redirect STDOUT though, if you want to prevent the parent process from hanging waiting for the child to finish. This is required cause exec will return the last line of the output of the child process, therefore, it needs to wait for the child's STDOUT to be closed.
That doesn't mean you need to redirect it to /dev/null. You can redirect it to some other file, or even to some other file descriptor (like STDERR: 1>&2).
exec('/path/to/executable'): will start a new process and wait for it to finish (i.e. blocking the parent process).
exec('/path/to/executable &'): basically the same as the above.
$pid = exec('/path/to/executable > /dev/null & echo $!'): will start a process in the background, with child and parent processes running in parallel. The output of /path/to/executable will be discarded, and $pid will receive the PID of the child process.
Using 2>&1 is actually not necessary, cause exec ignores the STDERR of the child process. It is also probably undesirable cause it will make it harder to find some errors, cause they will be just silently thrown away. If you omit 2>&1, you can pipe the STDERR of the parent process to some log file, that can be checked later when something goes wrong:
php /path/to/script.php 2>> /var/log/script_error.log
By using the above to start the script which triggers the child processes, everything that script.php and any child process write to STDERR will be written to the log file.
There are no threads in PHP. You could cheat by sending back an HTML page that triggers an Ajax call to start the heavy process in a new request. But if it's shared hosting, my guess is that you'll quickly hit the limits on memory, time or CPU usage imposed by your hosting provider.
$WshShell = new COM('WScript.Shell');
$oExec = $WshShell->Run('C:\xampp\php\php.exe C:\xampp\htdocs\test.php -a asdf', 0, true);
Cannot pass argv to test.php.
var_dump($argv);
Related
I have the following PHP 5.6.19 code on a Ubuntu 14.04 server. This code simply connects to a MySQL 5.6.28 database, waits a minute, launches another process of itself, then exits.
Note: this is the full script, and it's purpose is to demonstrate the problem - it doesn't do anything useful.
class DatabaseConnector {
const DB_HOST = 'localhost';
const DB_NAME = 'database1';
const DB_USERNAME = 'root';
const DB_PASSWORD = 'password';
public static $db;
public static function Init() {
if (DatabaseConnector::$db === null) {
DatabaseConnector::$db = new PDO('mysql:host=' . DatabaseConnector::DB_HOST . ';dbname=' . DatabaseConnector::DB_NAME . ';charset=utf8', DatabaseConnector::DB_USERNAME, DatabaseConnector::DB_PASSWORD);
}
}
}
$startTime = time();
// ***** Script works fine if this line is removed.
DatabaseConnector::Init();
while (true) {
// Sleep for 100 ms.
usleep(100000);
if (time() - $startTime > 60) {
$filePath = __FILE__;
$cmd = "nohup php $filePath > /tmp/1.log 2>&1 &";
// ***** Script sometimes exits here without opening the process and without errors.
$p = popen($cmd, 'r');
pclose($p);
exit;
}
}
I start the first process of the script using nohup php myscript.php > /tmp/1.log 2>&1 &.
This process loop should go on forever but... based on multiple tests, within a day (but not instantly), the process on the server "disappears" without reason. I discovered that the MySQL code is causing the popen code to fail (the script exits without any error or output).
What is happening here?
Notes
The server runs 24/7.
Memory is not an issue.
The database connects correctly.
The file path does not contain spaces.
The same problem exists when using shell_exec or exec instead of popen (and pclose).
I also know that popen is the line that fails because I did further debugging (not shown above) by logging to a file at certain points in the script.
Is the parent process definitely exiting after forking? I had thought pclose would wait for the child to exit before returning.
If it isn't exiting, I'd speculate that because the mySQL connection is never closed, you're eventually hitting its connection limit (or some other limit) as you spawn the tree of child processes.
Edit 1
I've just tried to replicate this. I altered your script to fork every half-second, rather than every minute, and was able to kill it off within about 10 minutes.
It looks like the the repeat creation of child processes is generating ever more FDs, until eventually it can't have any more:
$ lsof | grep type=STREAM | wc -l
240
$ lsof | grep type=STREAM | wc -l
242
...
$ lsof | grep type=STREAM | wc -l
425
$ lsof | grep type=STREAM | wc -l
428
...
And that's because the child's inheriting the parent's FDs (in this case for the mySQL connection) when it forks.
If you close the mySQL connection before popen with (in your case):
DatabaseConnector::$db = null;
The problem will hopefully go away.
I had a similar situation using pcntl_fork() and a MySQL connection. The cause here is probably the same.
Background info
popen() creates a child process. The call to pclose() closes the communication channel and the child process continues to run until it exits. This is when the things start to go out of control.
When a child process completes, the parent process receives a SIGCHLD signal. The parent process here is the PHP interpreter that runs the code you posted. The child process is the one launched using popen() (it doesn't matter what command it runs).
There is a small thing here you probably don't know or you have found in the documentation and ignored it because it doesn't make much sense when one programs in PHP. It is mentioned in the documentation of sleep():
If the call was interrupted by a signal, sleep() returns a non-zero value.
The sleep() PHP function is just a wrapper of the sleep() Linux system call (and usleep() PHP function is a wrapper of the usleep() Linux system call.)
What is not told in the PHP documentation is clearly stated in the documentation of the system calls:
sleep() makes the calling thread sleep until seconds seconds have elapsed or a signal arrives which is not ignored.
Back to your code.
There are two places in your code where the PHP interpreter calls the usleep() Linux system function. One of them is clearly visible: your PHP code invokes it. The other one is hidden (see below).
What happens (the visible part)
Starting with the second iteration, if a child process (created using popen() on a previous iteration) happens to exit while the parent program is inside the usleep(100000) call, the PHP interpreter process receives the SIGCHLD signal and its execution resumes before the time being out. The usleep() returns earlier than expected. Because the timeout is short, this effect is not observable by the naked eye. Put 10 seconds instead of 0.1 seconds and you'll notice it.
However, apart from the broken timeout, this doesn't affect the execution of your code in a fatal manner.
Why it crashes (the invisible part)
The second place where an incoming signal hurts your programs execution is hidden deep inside the code of the PHP interpreter. For some protocol reasons, the MySQL client library uses sleep() and/or usleep() in several places. If the interpreter happens to be inside one of these calls when the SIGCHLD arrives, the MySQL client library code is resumed unexpectedly and, many times, it concludes with the erroneous status "MySQL server has gone away (error 2006)".
It's possible that your code ignores (or swallows) the MySQL error status (because it doesn't expect it to happen in that place). Mine didn't and I spent a couple of days of investigation to find out the facts summarized above.
A solution
The solution for the problem is easy (after you know all the internal details exposed above). It is hinted in the documentation quote above: "a signal arrives which is not ignored".
The signals can be masked (ignored) when their arrival is not desired. The PHP PCNTL extension provides the function pcntl_sigprocmask(). It wraps the sigprocmask() Linux system call that sets what signals can be received by the program from now on (in fact, what signals to be blocked).
There are two strategies you can implement, depending of what you need.
If your program needs to communicate with the database and be notified when the child processed complete then you have to wrap all your database calls within a pair of calls to pcntl_sigprocmask() to block then unblock the SIGCHLD signal.
If you doesn't care when the child processes complete then you just call:
pcntl_sigprocmask(SIG_BLOCK, array(SIGCHLD));
before you start creating any child process (before the while()).
It makes your process ignore the termination of the child processes and lets it run its database queries without undesired interruption.
Warning
The default handling of the SIGCHLD signal is to call wait() in order to let the system cleanup after the completed child process. What happens if the signal is not handled (because its delivery is blocked) is explained in the documentation of wait():
A child that terminates, but has not been waited for becomes a "zombie". The kernel maintains a minimal set of information about the zombie process (PID, termination status, resource usage information) in order to allow the parent to later perform a wait to obtain information about the child. As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes. If a parent process terminates, then its "zombie" children (if any) are adopted by init(1), which automatically performs a wait to remove the zombies.
In plain English, if you block the reception of SIGCHLD signal, then you have to call pcntl_wait() in order to cleanup the zombie child processes.
You can add:
pcntl_wait($status, WNOHANG);
somewhere inside the while loop (just before it ends, for example).
the script exits without any error or output
Not surprising when there's no error checking in the code. However if it really is "crashing", then:
if the cause is trapped by the PHP runtime then it will be trying to log an error. Have you tried delibertely creating an error scenario to varify that the reorting/logging is working as you expect?
if the error is not trapped by the PHP runtime, the the OS should be dumping a corefile - have you checked the OS config? Looked for the core file? Analyzed it?
$cmd = "nohup php $filePath > /tmp/1.log 2>&1 &";
This probably doesn't do what you think it does. When you run a process in the background with most versions of nohup, it still retains a relationship with the parent process; the parent cannot be reaped until the child process exits - and a child is always spawning another child before it does.
This is not a valid way to keep your code running in the background / as a daemon. What the right approach is depends on what you are trying to achieve. Is there a specific reason for attempting to renew the process every 60 seconds?
(You never explicitly close the database connection - this is less of an issue as PHP should do this when exit is invoked).
You might want to read this and this
I suggest that process doesn't exit after pclose. In this case every process holds it's own connection to db. After some time connectons limit of MySQL is reached and new connection fails.
To understand what's going on - add some logs before and after strings DatabaseConnector::Init(); and pclose($p);
Specifics
I have an issue in PHP, when respawned processes are not handling signals, while before respawning, handling working correctly. I narrowed down my code to the very basic:
declare(ticks=1);
register_shutdown_function(function() {
if ($noRethrow = ob_get_contents()) {
ob_end_clean();
exit;
}
system('/usr/bin/nohup /usr/bin/php '.__FILE__. ' 1>/dev/null 2>/dev/null &');
});
function handler($signal)
{
switch ($signal) {
case SIGTERM:
file_put_contents(__FILE__.'.log', sprintf('Terminated [ppid=%s] [pid=%s]'.PHP_EOL, posix_getppid(), posix_getpid()), FILE_APPEND);
ob_start();
echo($signal);
exit;
case SIGCONT:
file_put_contents(__FILE__.'.log', sprintf('Restarted [ppid=%s] [pid=%s]'.PHP_EOL, posix_getppid(), posix_getpid()), FILE_APPEND);
exit;
}
}
pcntl_signal(SIGTERM, 'handler');
pcntl_signal(SIGCONT, 'handler');
while(1) {
if (time() % 5 == 0) {
file_put_contents(__FILE__.'.log', sprintf('Idle [ppid=%s] [pid=%s]'.PHP_EOL, posix_getppid(), posix_getpid()), FILE_APPEND);
}
sleep(1);
}
As you can see, it does following:
Registering shutdown function, in which respawn a process with nohup (so, to ignore SIGHUP when parent process dies)
Registering handler via pcntl_signal() for SIGTERM and SIGCONT. First will just log a message that process was terminated, while second will lead to respawn of the process. It is achieved with ob_* functions, so to pass a flag, what should be done in shutdown function - either exit or respawn.
Logging some information that script is "alive" to log file.
What is happening
So, I'm starting script with:
/usr/bin/nohup /usr/bin/php script.php 1>/dev/null 2>/dev/null &
Then, in log file, there are entries like:
Idle [ppid=7171] [pid=8849]
Idle [ppid=7171] [pid=8849]
Let's say, then I do kill 8849:
Terminated [ppid=7171] [pid=8849]
Thus, it is successful handling of SIGTERM (and script indeed exits). Now, if I instead do kill -18 8849, then I see (18 is numeric value for SIGCONT):
Idle [ppid=7171] [pid=8849]
Restarted [ppid=7171] [pid=8849]
Idle [ppid=1] [pid=8875]
Idle [ppid=1] [pid=8875]
And, therefore: first, SIGCONT was also handled correctly, and, judging by next "Idle" messages, newly spawned instance of script is working well.
Update #1 : I was thinking about stuff with ppid=1 (thus, init global process) and orphan processes signal handling, but it's not the case. Here is log part, which shows that orphan (ppid=1) process isn't the reason: when worker is started by controlling app, it also invokes it with system() command - same way like worker respawns itself. But, after controlling app invokes worker, it has ppid=1 and responds to signals correctly, while if worker respawns itself, new copy is not responding to them, except SIGKILL. So, issue appears only when worker respawns itself.
Update #2 : I tried to analyze what is happening with strace. Now, here are two blocks.
When worker was yet not respawned - strace output. Take a look on lines 4 and 5, this is when I send SIGCONT, thus kill -18 to a process. And then it triggers all the chain: writing to the file, system() call and exiting current process.
When worker was already respawned by itself - strace output. Here, take a look to lines 8 and 9 - they appeared after receiving SIGCONT. First of: looks like process is still somehow receiving a signal, and, second, it ignores the signal. No actions were done, but process was notified by the system that SIGCONT was sent. Why then the process ignores it - is the question (because, if installing of user handler for SIGCONT failed, then it should end execution, while process is not ended). As for SIGKILL, then output for already respawned worker is like:
nanosleep({1, 0}, <unfinished ...>
+++ killed by SIGKILL +++
Which indicates, that signal was received and did what it should do.
The problem
As the process is respawn, it is not reacting neither to SIGTERM, nor to SIGCONT. However, it is still possible to end it with SIGKILL (so, kill -9 PID indeed ends the process). For example, for process above both kill 8875 and kill -18 8875 will do nothing (process will ignore signals and continue to log messages).
However, I would not say that registering signals is failing completely - because it redefines at least SIGTERM (which normally leads to termination, while in this case it is ignored). Also I suspect that ppid = 1 points to some wrong thing, but I can not say for sure now.
Also, I tried any other kind of signals (in fact, it didn't matter what is the signal code, result was always the same)
The question
What could be the reason of such behavior? Is the way, which I'm respawning a process, correct? If not, what are other options which will allow newly spawned process to use user-defined signal handlers correctly?
Solution : Eventually, strace helped to understand the problem. This is as follows:
nanosleep({1, 0}, {0, 294396497}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
restart_syscall(<... resuming interrupted call ...>) = 0
Thus, it shows the signal was received, but ignored. To fully answer the question, I will need to figure out, why process added signals to ignore list, but unblocking them forcefully with pcntl_sigprocmask() is doing the thing:
pcntl_sigprocmask(SIG_UNBLOCK, [SIGTERM, SIGCONT]);
then all goes well and respawned process receives/handles signals as it is intended. I tried to add only SIGCONT for unblocking, for example - and then it was handled correctly, while SIGTERM was blocked, which points to the thing, that it is exactly the reason of failing to dispatch signals.
Resolution : for some reason, when process spawns itself with signal handlers installed, new instance has those signals masked for ignoring. Unmasking them forcefully solves the issue, but why signals are masked in new instance - that is an open question for now.
It's due to the fact, that you spawn a child process by executing system(foo), and then proceeding with dying of the current process. Hence, the process becomes an orphan, and its parent becomes PID 1 (init).
You can see the change using pstree command.
Before:
init─┬─cron
(...)
└─screen─┬─zsh───pstree
├─3*[zsh]
├─zsh───php
└─zsh───vim
After:
init─┬─cron
(...)
└─php
What wikipedia states:
Orphan processes is kind of the opposite situation of zombie processes, since it refers to the case where a parent process terminates before its child processes, in which case these children are said to become "orphaned".
Unlike the asynchronous child-to-parent notification that happens when a child process terminates (via the SIGCHLD signal), child processes are not notified immediately when their parent finishes. Instead, the system simply redefines the "parent-pid" field in the child process's data to be the process that is the "ancestor" of every other process in the system, whose pid generally has the value 1 (one), and whose name is traditionally "init". It is thus said that "init 'adopts' every orphan process on the system".
For your situation, I would suggest two options:
Use two scripts: one for managing the child, and second one, "worker", to actually perform the job,
or, use one script, that will include both: outer part will manage, inner part, forked from outer, will do the job.
I have a PHP site that performs Cron's that are triggered during client executions instead of a Cron manager. One of the Cron's that are performed takes a few seconds to execute, and it keeps the connection between the Client and the Server open until it is complete. Although I know I can set up a Cron to be fired from the Server instead of during Client runs, I would like to know if it is possible without following that format.
So, can the PHP script send a command to Apache (or whatever server it is hosted on) to close the connection between the Client and the Server, but continue to functions (so, without exiting)?
This works on Apache (and apparently not on IIS with FastCGI)
<?php
ignore_user_abort(true); // make sure PHP doesn't stop when the connection closes
// fire and forget - do lots of stuff so the connection actually closes
header("Content-Length: 0");
header("Connection: Close");
flush();
session_write_close(); // if you have a session
do_processing();
// don't forget to `set_time_limit` if your process takes a while
You can use shell to call the PHP binary... example:
shell("php -f /path/to/cronfile.php > /dev/null 2>/dev/null &"); which will run and not wait for a return.
See Asynchronous shell exec in PHP
Typically There is two forms of execution.
Client sided:
The client will remain on the page and the page will continue to process the commands given until completion.
Server Sided:
The client will navigate to a page & You make a switch in the database:
UPDATE Table SET PendinCron=1 WHERE IdentiferCol=$IdentiferData
Then you will have a Cronjob being run at X interval and will only process when the PendinCron is equal to one, if it is not. The Cron will not execute the required task.
I have one PHP script that has to start a second script in the background using exec. The parent script needs to capture the PID of this new process, but I also need to capture any potential output of the child script so it can be logged in the database. I can't just have the child script log to the database because I also need to capture fatal errors or anything that might indicate a problem with the script. The PID is used because the parent process needs to be able to check on the child and see when it finished. The child processes are sometimes run as crons, so forking isn't an option here either. I don't want two execution paths to debug if there are problems.
This was my first solution and it can capture the output, but fails to get the correct PID.
// RedirectToLog.php just reads stdin and logs it to the databse.
$cmd="php child.php </dev/null 2>&1 | php RedirectToLog.php >/dev/null 2>&1 & echo $!";
The problem here is that $! is the PID of the last process that was started in the background which ends up being RedirectToLog.php instead of child.php.
My other idea was to attempt to use a FIFO file (pipe).
$cmd1="php RedirectToLog.php </tmp/myFIFO >/dev/null 2>&1 &"
$cmd2="php child.php </dev/null >/tmp/myFIFO 2>&1 & echo $!"
This one didn't work because I couldn't get RedirectToLog to reliably consume the fifo and when it did, sometimes child.php failed to write EOF to the pipe which left both ends waiting on the other and both processes would hang until one was killed.
use proc_open to get full fd connectivity while starting a process. Take the resource returned by proc_open and use proc_get_status to get the pid.
$descriptors = array( ... ); // i suggest you read php.net on this
$pipes = array();
$res = proc_open($cmd,$descriptors,$pipes);
$info = proc_get_status($res);
echo $info['pid'];
I haven't fully grasped all your problems, but this should get you started.
I think you need to use
proc_open &
proc_get_status
I'm guessing the only reason you really want to capture the childs PID is to monitor it. If so, it might be easier to use proc_open. That way you can basically open file handles for stdin, stdout and stderr and monitor everything from the parent.
I am creating a process using proc_open in one PHP script.
How do i terminate this in another script . I am not able to pass the resource returned by the proc_open.
I also tried using proc_get_status() , it returns the ppid . I don't get the pid of the children .
development env : WAMP
Any inputs is appreciated .
I recommend that you re-examine your model to make certain that you actually have to kill the process from somewhere else. Your code will get increasingly difficult to debug and maintain in all but the most trivial circumstances.
To keep it encapsulated, you can signal the process you wish to terminate and gracefully exit in the process you want to kill. Otherwise, you can use normal IPC to send a message that says: "hey, buddy. shut down, please."
edit: for the 2nd paragraph, you may still end up launching a script to do this. that's fine. what you want to avoid is a kill -9 type of thing. instead, let the process exit gracefully.
To do that in pure PHP, here is the solution:
posix_kill($pid, 15); // SIGTERM = 15
You can use some methond to create process, this method usually returns the PID of the new process.
Does this works for You? :
$process = proc_open('php', $descriptorspec, $pipes, $cwd, $env);
$return_value = proc_close($process);
You're best off using something like this to launch your other process:
$pid = shell_exec("nohup $Command > /dev/null 2>&1 & echo $!");
That there would execute the process, and give you a running process ID.
exec("ps $pid", $pState);
$running = (count($pState) >= 2);
to terminate you can always use
exec("kill $pid");
However, you cant kill processes not owned by the user PHP runs at - if it runs as nobody - you'll start the new process as nobody, and only be able to kill processes running under the user nobody.