php5-fpm, killed by HUP Signal: Main Process failed to respawn - php

Question

I had a problem on my main php server, wherein the main php5-fpm process would be killed by an HUP signal. After the main process would be killed it would fail to respawn. Since each child process is allowed only to server a certain number of requests, they would eventually die without spawning any other child process. This would cause the server to die and my users would receive a 502 response from the server. I was initially able to solve this issue by have a cron that would check the thread count of PHP processes and then restart if its less than 5.
Sep 14 11:41:41 ubuntu kernel: [ 3699.092724] init: php5-fpm main process (3592) killed by HUP signal
Sep 14 11:41:41 ubuntu kernel: [ 3699.092740] init: php5-fpm main process ended, respawning
Sep 14 11:41:42 ubuntu kernel: [ 3699.160940] init: php5-fpm main process (3611) terminated with status 78
Sep 14 11:41:42 ubuntu kernel: [ 3699.160954] init: php5-fpm main process ended, respawning
Sep 14 11:41:42 ubuntu kernel: [ 3699.216950] init: php5-fpm main process (3619) terminated with status 78
Sep 14 11:41:42 ubuntu kernel: [ 3699.216966] init: php5-fpm main process ended, respawning
Sep 14 11:41:42 ubuntu kernel: [ 3699.283573] init: php5-fpm main process (3627) terminated with status 78
Sep 14 11:41:42 ubuntu kernel: [ 3699.283590] init: php5-fpm main process ended, respawning
Sep 14 11:41:42 ubuntu kernel: [ 3699.337563] init: php5-fpm main process (3635) terminated with status 78
Sep 14 11:41:42 ubuntu kernel: [ 3699.337579] init: php5-fpm main process ended, respawning
Sep 14 11:41:42 ubuntu kernel: [ 3699.385293] init: php5-fpm main process (3643) terminated with status 78
Sep 14 11:41:42 ubuntu kernel: [ 3699.385305] init: php5-fpm main process ended, respawning
Sep 14 11:41:42 ubuntu kernel: [ 3699.430903] init: php5-fpm main process (3651) terminated with status 78
Sep 14 11:41:42 ubuntu kernel: [ 3699.430913] init: php5-fpm main process ended, respawning
Sep 14 11:41:42 ubuntu kernel: [ 3699.482790] init: php5-fpm main process (3659) terminated with status 78
Sep 14 11:41:42 ubuntu kernel: [ 3699.482800] init: php5-fpm main process ended, respawning
Sep 14 11:41:42 ubuntu kernel: [ 3699.532239] init: php5-fpm main process (3667) terminated with status 78
Sep 14 11:41:42 ubuntu kernel: [ 3699.532249] init: php5-fpm main process ended, respawning
Sep 14 11:41:42 ubuntu kernel: [ 3699.595810] init: php5-fpm main process (3675) terminated with status 78
Sep 14 11:41:42 ubuntu kernel: [ 3699.595825] init: php5-fpm main process ended, respawning
Sep 14 11:41:42 ubuntu kernel: [ 3699.648253] init: php5-fpm main process (3683) terminated with status 78
Sep 14 11:41:42 ubuntu0 kernel: [ 3699.648265] init: php5-fpm respawning too fast, stopped
My upstart script config
# php5-fpm - The PHP FastCGI Process Manager
description "The PHP FastCGI Process Manager"
author "Ondřej Surý <ondrej#debian.org>"
start on runlevel [2345]
stop on runlevel [016]
# Precise upstart does not support reload signal, and thus rejects the
# job. We'd rather start the daemon, instead of forcing users to
# reboot https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1272788
#
#reload signal USR2
pre-start exec /usr/lib/php5/php5-fpm-checkconf
respawn
exec /usr/sbin/php5-fpm --nodaemonize --fpm-config /etc/php5/fpm/php-fpm.conf

Answer 1

After searching the internet was finally able to get a solution to this by modifying the upstart script of php5-fpm in /etc/init/php5-fpm.conf
# php5-fpm - The PHP FastCGI Process Manager
description "The PHP FastCGI Process Manager"
author "Ondřej Surý <ondrej#debian.org>"
start on runlevel [2345]
stop on runlevel [016]
# Precise upstart does not support reload signal, and thus rejects the
# job. We'd rather start the daemon, instead of forcing users to
# reboot https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1272788
#
#reload signal USR2
pre-start exec /usr/lib/php5/php5-fpm-checkconf
pre-start exec /bin/bash /etc/init/php5-fpm.sh
post-start exec /bin/bash /etc/init/php5-fpm-onstart.sh
respawn
exec /usr/sbin/php5-fpm --nodaemonize --fpm-config /etc/php5/fpm/php-fpm.conf
So added additional scripts pre-start and post-start in the php5-fpm.conf. The pre-start script is
#!/bin/bash
rm /var/run/php5-fpm.pid
rm /var/run/php5-fpm.sock
CHILD_PIDS_FILE="/var/run/php5-fpm-child.pid"
CHILD_PIDS=`ps -ef | grep 'php' | grep -v grep |awk '{print $2}'`
echo "$CHILD_PIDS" > "$CHILD_PIDS_FILE"
The script basically deletes the main process pid and the sock file. Then writes the pids of the child processes to the file so than they can be killed once a new php5-fpm process is created.
The post-start script is
#!/bin/bash
CHILD_PIDS_FILE="/var/run/php5-fpm-child.pid"
while read PID; do
kill -9 $PID
done < $CHILD_PIDS_FILE
>$CHILD_PIDS_FILE
The post-start script deletes all the child-pids that were running before php5-fpm restarted.