I'm working on a pre-forking TCP socket server written in PHP.
The daemon (the parent process), forks some number of children and then waits until it's told to exit and the children are all gone or it receives a signal.
SIGINT and SIGTERM cause it to send SIGTERM to all of the children.
The children set up their own signal handlers: SIGTERM causes a clean exit. SIGUSR1 causes it to dump some status information (just print that it received the signal in the sample code below).
If a child exits unexpectedly, the parent starts a new child unless the exiting flag has been set by the SIGINT handler.
The initial children, forked during daemon initialization, react to signals as expected.
Newly forked children to replace an unexpected child exit, do not respond to signals.
The following code can be used to demonstrate this:
<?php
$children = [];
$exiting = false;
pcntl_async_signals( true );
pcntl_signal( SIGCHLD, 'sigchldHandler' );
pcntl_signal( SIGINT, 'sigintHandler' );
pcntl_signal( SIGTERM, 'sigintHandler' );
// Fork our children.
for( $ii = 0; $ii < 1; $ii++ )
{
startChild();
}
// Forks a single child.
function startChild()
{
global $children;
echo "Parent: starting child\n";
$pid = pcntl_fork();
switch( true )
{
case ( $pid > 0 ):
$children[$pid] = $pid;
break;
case ( $pid === 0 ):
child();
exit( 0 );
default:
die( 'Parent: pcntl_fork() failed' );
break;
}
}
// As long as we have any children...
while( true )
{
if( empty( $children ) ) break;
sleep( 1 );
}
// The child process.
function child()
{
$pid = posix_getpid();
echo "Child $pid: started\n";
sleep( 10 ); // Give us a chance to start strace (4/30/19 08:27)
pcntl_sigprocmask( SIG_SETMASK, [] ); // Make sure nothing is blocked.
pcntl_async_signals( true ); // This may be inherited.
pcntl_signal( SIGINT, SIG_IGN ); // Ignore SIGINT.
pcntl_signal( SIGTERM, function() use ( $pid ) // Exit on SIGTERM.
{
echo "Child $pid: received SIGTERM\n";
exit( 0 );
}, false );
pcntl_signal( SIGUSR1, function() use( $pid ) // Acknowledge SIGUSR1.
{
printf( "Child %d: Received SIGUSR1\n", $pid );
});
// Do "work" here.
while( true )
{
sleep( 60 );
}
}
// Handle SIGCHLD in the parent.
// Start a new child unless we're exiting.
function sigchldHandler()
{
global $children, $exiting;
echo "Parent: received SIGCHLD\n";
while( true )
{
if( ( $pid = pcntl_wait( $status, WNOHANG ) ) < 1 )
{
break;
}
echo "Parent: child $pid exited\n";
unset( $children[$pid] );
if( !$exiting )
{
startChild();
}
}
}
// Handle SIGINT in the parent.
// Set exiting to true and send SIGTERM to all children.
function sigintHandler()
{
global $children, $exiting;
$exiting = true;
echo PHP_EOL;
foreach( $children as $pid )
{
echo "Parent: sending SIGTERM to $pid\n";
posix_kill( $pid, SIGTERM );
}
}
Run this script in a terminal session. The initial output will be similar to this with a different PID:
Parent: starting child
Child 65016: started
From a different terminal session issue a kill command:
# kill -USR1 65016
The child process will display this in the first terminal session:
Child 65016: Received SIGUSR1
The child is receiving and processing signals as expected. Now terminate that first child:
# kill -TERM 65016
The output to the first terminal session will look like this (with different PIDS):
Child 65016: received SIGTERM
Parent: received SIGCHLD
Parent: child 65016 exited
Parent: starting child
Child 65039: started
The new child process will receive but react to any signals at this point except SIGKILL and SIGSTOP which can't be caught.
Sending the parent a SIGINT will cause it to send a SIGTERM to the new child. The child won't get it and parent will wait until the child is forcibly killed before exiting (yes, the production code will include a timeout and SIGKILL any remaining children).
Environment:
- Ubuntu 18.04.2
- macOS Mojave 10.14.3 (same behavior)
- PHP 7.2.17 (cli)
I find myself out of ideas. Thoughts?
EDIT 30-Apr-2019 08:27 PDT:
I have a little more information. I added a sleep( 10 ) right after the 'echo "Child $pid: started\n";' to give me a chance to run strace on the child.
Based on the strace output, it looks like the signals are being delivered, but the child signal handler is not called.
# sudo strace - p 69710
strace: Process 69710 attached
restart_syscall(<... resuming interrupted nanosleep ...>) = 0
rt_sigprocmask( SIG_SETMASK, [], ~[ KILL STOP RTMIN RT_1], 8) = 0
rt_sigaction( SIGINT, {sa_handler = SIG_IGN, sa_mask = [], sa_flags = SA_RESTORER, sa_restorer = 0x7f6e8881cf20}, null, 8) = 0
rt_sigprocmask( SIG_UNBLOCK, [ INT ], null, 8 ) = 0
rt_sigaction( SIGTERM, {sa_handler = 0x55730bdaf2e0, sa_mask = ~[ ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags = SA_RESTORER | SA_INTERRUPT | SA_SIGINFO, sa_restorer = 0x7f6e8881cf20}, null, 8) = 0
rt_sigprocmask( SIG_UNBLOCK, [ TERM ], null, 8 ) = 0
rt_sigaction( SIGUSR1, {sa_handler = 0x55730bdaf2e0, sa_mask = ~[ ILL TRAP ABRT BUS FPE KILL SEGV CONT STOP TSTP TTIN TTOU SYS RTMIN RT_1], sa_flags = SA_RESTORER | SA_RESTART | SA_SIGINFO, sa_restorer = 0x7f6e8881cf20}, null, 8) = 0
rt_sigprocmask( SIG_UNBLOCK, [ USR1 ], null, 8 ) = 0
nanosleep({tv_sec = 60, tv_nsec = 0}, 0x7ffe79859470) = 0
nanosleep({tv_sec = 60, tv_nsec = 0}, {tv_sec = 37, tv_nsec = 840636107}) = ? ERESTART_RESTARTBLOCK( Interrupted by signal)
--- SIGUSR1 {si_signo = SIGUSR1, si_code = SI_USER, si_pid = 69544, si_uid = 1000} ---
rt_sigreturn({mask = []}) = -1 EINTR( Interrupted system call)
rt_sigprocmask( SIG_BLOCK, ~[ RTMIN RT_1], [], 8) = 0
rt_sigprocmask( SIG_SETMASK, [], null, 8 ) = 0
nanosleep({tv_sec = 60, tv_nsec = 0}, 0x7ffe79859470) = 0
I believe the problem is PHP signal handling doesn't work as one may intend to when pcntl_fork is called inside of a registered signal handling function. Since the second child process is created inside of sigchldHandler it won't receive process subsequent signals.
Edit: Unfortunately I don't have any references for this. I've been bashing my head against the wall myself with a similar problem as OP (hence the new account!) and I can't find any definitive answers or explanations for this behavior, just the evidence from manual stub tests. I'd love to know as well (:
Related
I updated to PHP 7 at my localhost, but since then anytime i want to redirect from one page to another in my nette application, I'll receive error: 500 - Internal Server Error.
I was searching through stack overflow and found a problem that is quite similar to mine here: How to solve "mod_fastcgi.c.2566 unexpected end-of-file (perhaps the fastcgi process died)" when calling .php that takes long time to execute? . However, I don't work with large files and my connection dies immediately.
My /var/log/lighttpd/error.log
2016-03-06 10:54:11: (server.c.1456) [note] graceful shutdown started
2016-03-06 10:54:11: (server.c.1572) server stopped by UID = 0 PID = 351
2016-03-06 11:03:48: (log.c.194) server started
2016-03-06 11:07:17: (mod_fastcgi.c.2390) unexpected end-of-file (perhaps the fastcgi process died): pid: 21725 socket: unix:/run/lighttpd/php-fastcgi.sock-3
2016-03-06 11:07:17: (mod_fastcgi.c.3171) response not received, request sent: 1029 on socket: unix:/run/lighttpd/php-fastcgi.sock-3 for /~rost/lp/web/www/index.php?, closing connection
2016-03-06 11:09:01: (mod_fastcgi.c.2390) unexpected end-of-file (perhaps the fastcgi process died): pid: 21725 socket: unix:/run/lighttpd/php-fastcgi.sock-3
2016-03-06 11:09:01: (mod_fastcgi.c.3171) response not received, request sent: 1061 on socket: unix:/run/lighttpd/php-fastcgi.sock-3 for /~rost/lp/web/www/index.php?action=list&presenter=Campaign, closing connection
2016-03-06 11:09:06: (mod_fastcgi.c.2390) unexpected end-of-file (perhaps the fastcgi process died): pid: 21725 socket: unix:/run/lighttpd/php-fastcgi.sock-3
2016-03-06 11:09:06: (mod_fastcgi.c.3171) response not received, request sent: 942 on socket: unix:/run/lighttpd/php-fastcgi.sock-3 for /~rost/lp/web/www/index.php?, closing connection
2016-03-06 11:09:14: (mod_fastcgi.c.2390) unexpected end-of-file (perhaps the fastcgi process died): pid: 21725 socket: unix:/run/lighttpd/php-fastcgi.sock-3
2016-03-06 11:09:14: (mod_fastcgi.c.3171) response not received, request sent: 1051 on socket: unix:/run/lighttpd/php-fastcgi.sock-3 for /~rost/lp/web/www/index.php?action=out&presenter=Sign, closing connection
My /etc/lighttpd/lighttpd.conf
server.modules = ( "mod_userdir",
"mod_access",
"mod_accesslog",
"mod_fastcgi",
"mod_rewrite",
"mod_auth"
)
server.port = 80
server.username = "http"
server.groupname = "http"
server.document-root = "/srv/http"
server.errorlog = "/var/log/lighttpd/error.log"
dir-listing.activate = "enable"
index-file.names = ( "index.html" )
# Rewrite URL without dots to index.php
#url.rewrite-once = ( "/^[^.?]*$/" => "/index.php" )
mimetype.assign = ( ".html" => "text/html",
".htm" => "text/html",
".txt" => "text/plain",
".properties" => "text/plain",
".jpg" => "image/jpeg",
".png" => "image/png",
".svg" => "image/svg+xml",
".gif" => "image/gif",
".css" => "text/css",
".js" => "application/x-javascript",
"" => "application/octet-stream"
)
userdir.path = "public_html"
# Fast CGI
include "conf.d/fastcgi.conf"
My /etc/lighttpd/conf.d/fastcgi.conf
server.modules += ( "mod_fastcgi" )
#server.indexfiles += ( "index.php" ) #this is deprecated
index-file.names += ( "index.php" )
fastcgi.server = (
".php" => (
"localhost" => (
"bin-path" => "/usr/bin/php-cgi",
"socket" => "/run/lighttpd/php-fastcgi.sock",
"max-procs" => 4, # default value
"bin-environment" => (
"PHP_FCGI_CHILDREN" => "1", # default value
),
"broken-scriptfilename" => "enable"
))
)
Variables from /etc/php/php.ini
cat /etc/php/php.ini | grep max_execution_time
max_execution_time = 30
cat /etc/php/php.ini | grep default_socket_timeout
default_socket_timeout = 60
Update 7.3.2016
I switched from php fast cgi to php-fpm and interesting thing is that problem prevails, but is less often. Sometimes the redirect jump to 500 and sometimes not. And error log again:
2016-03-07 22:23:32: (mod_fastcgi.c.2390) unexpected end-of-file (perhaps the fastcgi process died): pid: 0 socket: unix:/run/php-fpm/php-fpm.sock
2016-03-07 22:23:32: (mod_fastcgi.c.3171) response not received, request sent: 1084 on socket: unix:/run/php-fpm/php-fpm.sock for /~rost/lp/web/www/index.php?action=out&presenter=Sign, closing connection
Also, try to delete Nette cache first.
I've finally found a solution. It is probably Nette / Cassandra related problem.
The error was appearing because of object Nette\Security\Identity, after I assigned user data into it:
public function authenticate(array $credentials) {
// Retrieve username and password
list($email, $passwd) = $credentials;
// Select user with given email from database
$usr = $this->daoManager->getDao("AppUser")->loadByEmail($email);
if ($usr == null || count($usr) == 0) {
$msg = 'The email is incorrect.';
$arg = self::IDENTITY_NOT_FOUND;
throw new Nette\Security\AuthenticationException($msg, $arg);
}
// TODO Check user password
// TODO Check verification
// Create identity - THE PROBLEM WAS HERE
return new Identity($email, $usr['role'], $usr);
}
It was caused by value 'registered' in $usr array which was of type Cassandra\Timestamp. Since then almost every redirect crashed with above mentioned error.
Following code fixed the issue:
return new Identity($email, $usr['role'], $this->fixUserArray($usr));
Where:
protected function fixUserArray(array $user) {
$result = array();
foreach ($user as $key => $val) {
if ($key === "registered") {
$result[$key] = $val->time();
} else {
$result[$key] = $val;
}
}
return $result;
}
I already have a function on a website for a client that grabs data from a live server to save it locally, but something I didn't expect was that sometimes these local servers aren't in an area with good service, so every now and then the script dies after a certain amount of time because it failed to connect.
I already have a system implemented to disable external calls for these types of situations, but the clients can't get to the option to set this "Offline Mode" to begin with because the service is bad and the server is trying to reach the live server.
So what I need to do, is wrap my SyncTable function with a function like set_time_limit(8) that calls a different function that sets the "Offline Mode" automatically if the SyncTable Function fails to complete in the 8 seconds.
Is something like this possible? If so, I would love to know how so I can save these clients some time in areas with rough service.
You can accomplish this using proc_open, proc_get_status, and proc_terminate to start the SyncTable operation as a process, monitor it, and terminate it if necessary. Note: You may need to create a simple wrapper script so you can start the SyncTable function as a standalone process.
Here's a function I use to do this and enforce a timeout:
/// Executes a command and returns the output
/// If the timeout value (in seconds) is reached, it terminates the process
/// and returns FALSE
function exec_timeout($cmd, $timeout=30)
{
$descriptors = array(
0 => array('pipe', 'r'), // stdin
1 => array('pipe', 'w'), // stdout
2 => array('pipe', 'w') // stderr
);
$pipes = Array();
$process = proc_open($cmd, $descriptors, $pipes);
$result = '';
$end_time = time() + $timeout;
if (is_resource($process))
{
// set the streams to non-blocking
stream_set_blocking($pipes[0], 0);
stream_set_blocking($pipes[1], 0);
stream_set_blocking($pipes[2], 0);
$timeleft = $end_time - time();
while ($timeleft > 0)
{
$status = proc_get_status($process);
$result .= stream_get_contents($pipes[1]);
// leave the loop if the process has already finished
if (!$status['running'])
break;
$timeleft = $end_time - time();
}
if ($timeleft <= 0)
{
proc_terminate($process);
$result = FALSE;
}
}
// check for errors
$errors = stream_get_contents($pipes[2]);
if (!empty($errors))
fwrite(STDERR, "$errors\n");
// close streams
fclose($pipes[0]);
fclose($pipes[1]);
fclose($pipes[2]);
proc_close($process);
return $result;
}
I want to check to see if Gearman daemon is running. And only then run tasks so my application does not crash.
Here's my code:
$daemonRunning = true;
while( true )
{
try
{
Yii::app()->gearman->client->ping( true );
if ( $daemonRunning === false )
{
echo "Daemon back online. Starting signature process...\n";
}
Yii::app()->gearman->client->runTasks();
}
catch( GearmanException $e )
{
echo "Daemon appears to be down. Waiting for it to come back up...\n";
$daemonRunning = false;
}
sleep(1);
}
But the issue is that ping does not throw an exception, it throws a fatal error:
PHP Error[2]: GearmanClient::ping(): flush(GEARMAN_COULD_NOT_CONNECT) 127.0.0.1:4730 -> libgearman/connection.cc:673
Though oddly, if I remove ping, and use only runTasks, then an exception is thrown.
Related:
How do I handle the error when Gearman daemon goes down while processes are running? I get the following error from PHP when I bring down the Gearman daemon:
php: libgearman/universal.cc:481: gearman_return_t connection_loop(gearman_universal_st&, const gearman_packet_st&, Check&): Assertion `&con->_packet == universal.packet_list' failed.
Aborted (core dumped)
At it's most basic, checking the status of a Gearman server can be done through command line using:
(echo status ; sleep 0.1) | nc 127.0.0.1 4730 -w 1
However as noted in this question, you can use fsocketopen to get he status of the gearman server.
// Taken from https://stackoverflow.com/questions/2752431/any-way-to-access-gearman-administration
class Waps_Gearman_Server {
/**
* #var string
*/
protected $host = "127.0.0.1";
/**
* #var int
*/
protected $port = 4730;
/**
* #param string $host
* #param int $port
*/
public function __construct($host=null,$port=null){
if( !is_null($host) ){
$this->host = $host;
}
if( !is_null($port) ){
$this->port = $port;
}
}
/**
* #return array | null
*/
public function getStatus(){
$status = null;
$handle = fsockopen($this->host,$this->port,$errorNumber,$errorString,30);
if($handle!=null){
fwrite($handle,"status\n");
while (!feof($handle)) {
$line = fgets($handle, 4096);
if( $line==".\n"){
break;
}
if( preg_match("~^(.*)[ \t](\d+)[ \t](\d+)[ \t](\d+)~",$line,$matches) ){
$function = $matches[1];
$status['operations'][$function] = array(
'function' => $function,
'total' => $matches[2],
'running' => $matches[3],
'connectedWorkers' => $matches[4],
);
}
}
fwrite($handle,"workers\n");
while (!feof($handle)) {
$line = fgets($handle, 4096);
if( $line==".\n"){
break;
}
// FD IP-ADDRESS CLIENT-ID : FUNCTION
if( preg_match("~^(\d+)[ \t](.*?)[ \t](.*?) : ?(.*)~",$line,$matches) ){
$fd = $matches[1];
$status['connections'][$fd] = array(
'fd' => $fd,
'ip' => $matches[2],
'id' => $matches[3],
'function' => $matches[4],
);
}
}
fclose($handle);
}
return $status;
}
}
In regards to your second question, I have never been able to recover a gearman worker when the connect is lost midwork. You basically have to kill the entire process running the client worker, and let supervisor take back over and restart another worker process.
Gearman client workers should be extremeley ephemeral. You should be monitoring them for memory usage regularly and auto killing them. So, if you ever hit a segfault/coredump, killing the worker is totally normal.
As mentioned, you can auto start back up your workers using Supervisor. Also check this out, it explains how to get Gearman Clients working with Supervisor
We're using a modified version of easy-apns for sending our push messages. With the enhanced push message format the apple server responds if an error occurs, and does nothing if everything goes well.
The problem is that we have to wait for an error a certain amount of time after each message has been sent. For example if we receive no response after 1 second, we assume everything went ok.
With 20000 push messages, this takes far too long. Is there any way I can listen for errors in a faster way? For example sending to 1000 devices and then listen for errors? What happens if the connection gets closed, can I still read the error response?
Ideal would be some kind of asynchronous writing and reading, but I think that's not possible.
Here's the corresponding code:
$fp = $this->connect();
$expiry = time()+60*60;
// construct message
$msg = chr(1).pack("N",$batchid).pack("N",$expiry).pack("n",32).pack('H*',$devicetoken).pack("n",strlen($payload)).$payload;
// send message to Apple
$fwrite = fwrite($fp, $msg);
if(!$fwrite) {
// connection has been closed
$this->disconnect();
throw new Exception("Connection closed");
} else {
// read response from Apple
// Timeout. 1 million micro seconds = 1 second
$tv_sec = 1;
$tv_usec = 0;
$r = array($fp);
$we = null; // Temporaries. "Only variables can be passed as reference."
// PROBLEM: this method waits for $tv_sec seconds for a response
$numChanged = stream_select($r, $we, $we, $tv_sec, $tv_usec);
if( $numChanged === false ) {
throw new Exception("Failed selecting stream to read.");
} elseif ( $numChanged > 0 ) {
$command = ord( fread($fp, 1) );
$status = ord( fread($fp, 1) );
$identifier = implode('', unpack("N", fread($fp, 4)));
if( $status > 0 ) {
// The socket has also been closed. Cause reopening in the loop outside.
$this->disconnect();
throw new MessageException("APNS responded with status $status: {$this->statusDesc[$status]} ($devicetoken).".microtime(), $status);
} else {
// unknown response, assume ok
}
} else {
// no response, assume ok
}
}
i was programming a mail templatingsystem. The user should be able to use markers in there, they will be replaced by the actual data. The problem ist, my function to replace the markers works just fine, but i need to do a recursiv call of that function, that will only run once, and this is what i came up with:
public function replace_placeholders($content, $recipient, $settings, $interface, $recommendation, $format, $recursion = false) {
$content = $this->replace_ph('briefanrede' , $recipient['id'] , $content);
$content = $this->replace_ph('anrede' , $recipient['title'] , $content);
$content = $this->replace_ph('email' , $recipient['email'] , $content);
$content = $this->replace_ph('kundennummer' , $recipient['kdnumber'] , $content);
$content = $this->replace_ph('briefanrede' , $recipient['briefanrede'] , $content);
if($recipient['title'] == $settings['anrede_w'] || $recipient['title'] == $settings['anrede_m']) {
$content = $this->replace_ph('vorname' , $recipient['forename'] , $content);
$content = $this->replace_ph('nachname' , $recipient['surename'] , $content);
} else {
$content = $this->replace_ph('vorname' , "" , $content, true);
$content = $this->replace_ph('nachname' , "" , $content, true);
}
$content = $this->replace_salutation($recipient, $settings, $content);
//Recommendation
if($this->need_replacement($content, 'weiterempfehlung') === false && $recursion === false) {
if($recommendation['own_page'] == 1) {
$baseurl = $recommendation['location'];
} else {
$baseurl = $recommendation['link'];
}
$pattern = ($format == "html") ? '%s' : '%s';
$url = $this->replace_placeholders($baseurl, $recipient, $settings, $interface, $recommendation, true);
$content = $this->replace_ph('weiterempfehlung' , (($format == "html") ? sprintf($pattern, $url, $settings['text_weiterempfehlung']): sprinf($pattern, $url)), $content);
}
return $content;
}
The recursiv call in this line
$url = $this->replace_placeholders($baseurl, $recipient, $settings, $interface, $recommendation, true);
is causing a 500 internal server error. I dont know why, because i think that i limited the recursion to run once. Can you help me out?
Sorry for my bad english i try hard to write clear sentences.
//EDIT:
Apache log:
[Wed May 30 15:31:56 2012] [warn] [client xx.xxx.xx.xxx] (104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server
[Wed May 30 15:31:56 2012] [warn] [client xx.xxx.xx.xxx] (104)Connection reset by peer: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[Wed May 30 15:31:56 2012] [error] [client xx.xxx.xx.xxx] File does not exist: /var/www/web80/html/web80-newsletter/favicon.ico
[Wed May 30 15:31:58 2012] [error] mod_fcgid: process /var/www/php-fcgi/web80.php53/php-fcgi(21975) exit(communication error), get unexpected signal 11
the php errorlog is empty.
It would seem you miss one argument in your recursive call, making the $recursive = false continue being false all the time, which in turn makes your if statement
if($this->need_replacement($content, 'weiterempfehlung') === false && $recursion === false)
always return true.
Try adding one last variable to your recursive call instead and you should be able to properly execute your script, ie:
$url = $this->replace_placeholders($baseurl, $recipient, $settings, $interface,
$recommendation, true, true);
^ added one true
What i think you want to add instead of the first true is $format.
Signal 11 is SIGSEGV, i.e. the process crashed due to a bad memory access (such as dereferencing a NULL pointer or accessing memory it was not supposed to access).
This is nothing a PHP script should be causing, so you should first upgrade to the most recent stable PHP version and if it still happens reduce your script as much as possible (remove everything that can be removed while the crash still happens) and then report it as a PHP bug.