php-fpm child process exited on signal 11 - php

Our application runs in a Docker container on AWS:
Operating system: Ubuntu 14.04.2 LTS (Trusty Tahr)
Nginx version: nginx/1.4.6 (Ubuntu)
Memcached version: memcached 1.4.14
PHP version: PHP 5.5.9-1ubuntu4.11 (cli) (built: Jul 2 2015 15:23:08)
System Memory: 7.5 GB
We get blank pages and a 404 Error less frequently. While checking the logs, I found that the php-child process is killed and it seems that memory is mostly used by memcache and php-fpm process and very low free memory.
memcache is configured to use 2 GB memory.
Here is php www.conf
pm = dynamic
pm.max_children = 30
pm.start_servers = 9
pm.min_spare_servers = 4
pm.max_spare_servers = 14
rlimit_files = 131072
rlimit_core = unlimited
Error logs
/var/log/nginx/php5-fpm.log
[29-Jul-2015 14:37:09] WARNING: [pool www] child 259 exited on signal 11 (SIGSEGV - core dumped) after 1339.412219 seconds from start
/var/log/nginx/error.log
2015/07/29 14:37:09 [error] 141#0: *2810 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: x.x.x.x, server: _, request: "GET /suggestions/business?q=Selectfrom HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "http://example.com/"
/var/log/nginx/php5-fpm.log
[29-Jul-2015 14:37:09] NOTICE: [pool www] child 375 started
/var/log/nginx/php5-fpm.log:[29-Jul-2015 14:37:56] WARNING: [pool www] child 290 exited on signal 11 (SIGSEGV - core dumped) after 1078.606356 seconds from start
Coredump
Core was generated by php-fpm: pool www.Program terminated with signal SIGSEGV, Segmentation fault.#0 0x00007f41ccaea13a in memcached_io_readline(memcached_server_st*, char*, unsigned long, unsigned long&) () from /usr/lib/x86_64-linux-gnu/libmemcached.so.10
dmesg
[Wed Jul 29 14:26:15 2015] php5-fpm[12193]: segfault at 7f41c9e8e2da ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:28:26 2015] php5-fpm[12211]: segfault at 7f41c966b2da ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:29:16 2015] php5-fpm[12371]: segfault at 7f41c9e972da ip 00007f41ccaea13a sp 00007ffcc5730b70 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:35:36 2015] php5-fpm[12469]: segfault at 7f41c96961e9 ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:35:43 2015] php5-fpm[12142]: segfault at 7f41c9e6c2bd ip 00007f41ccaea13a sp 00007ffcc5730b70 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:37:07 2015] php5-fpm[11917]: segfault at 7f41c9dd22bd ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:37:54 2015] php5-fpm[12083]: segfault at 7f41c9db72bd ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]

While googling for this same issue, and trying hard to find a solution that was not related to sessions (because I have ruled that out) nor to bad PHP code (because I have several websites running precisely the same version of WordPress, and none have issues... except for one), I came upon an answer telling that a possible solution did involve removing some buggy extension (usually memcache/d, but could be something else).
Since I had this same site working flawlessly on one Ubuntu server, when switching to a newer server, I immediately suspected that it was the migration from PHP 5.5 to 7 that caused the problem. It was just strange because no other website was affected. Then I remembered that another thing was different on this new server: I had also installed New Relic. This is both an extension and a small server that runs in the background and sends a lot of analytics data to New Relic for processing. Allegedly, it's a PHP 5 extension, but, surprisingly, it loads well on PHP 7, too.
Now here comes the tricky bit. At some point, I had installed W3 Total Cache for the WordPress installation of that particular website. Subsequently, I saw that the performance of that server was so stellar that W3TC was unnecessary, and simply stuck to a much simpler configuration. So I could uninstall W3TC. That's all very nice, but... I forgot that I had turned New Relic on W3TC, too (allegedly, it adds some extra analytics data to be sent to New Relic). When uninstalling W3TC, probably there was 'something' left on the New Relic configuration in my server which was still attempting to send data through the W3TC interface (assuming that W3TC has an interface... I really have no idea how it works at that level), and, because that specific bit of code was missing, the php_fpm handler for that website would fail... some of the time. Not all the time, because I'm assuming that, in most cases, nginx was sending static pages back. Or maybe php_fpm, set to 'recycle' after 100 calls or so, would crash-on-stop. Whatever exactly was happening, it was definitely related to New Relic — as soon as I removed the New Relic extension from PHP, that website went back to working normally.
Because this is such a specific scenario, I'm just writing this as an answer, in the remote chance that someone in the future googles for the exact problem.

In my case it was related to zend debug/xdebug. It forwards some TCP packets to the IDE (PhpStorm), that was not listening on this port (debugging was off). The solution is to either disable these extensions or enable debug listening on the debugging port.

I had this problem after installing xdebug, adding some properties to /etc/php/7.1/fpm/php.ini and restarting nginx. This is running on a Homestead Laravel box.
Simply restarting the php7.1-fpm service solved it for me.

It can happen if PHP is unable to write the session information to a file. By default it is /var/lib/php/session. You can change it by using configuration session_save_path.
phpMyAdmin having problems on nginx and php-fpm on RHEL 6

In my case it was Xdebug. After uninstalling it, it got back to normal.

In my case, it was caused by the New Relic PHP Agent. Therefore, for a specific function that caused a crash, I added this code to disable New Relic:
if (function_exists('newrelic_ignore_transaction')) {
newrelic_ignore_transaction();
}
Refer to: https://discuss.newrelic.com/t/how-to-disable-a-specific-transaction-in-php-agent/42384/2

In our case it was caused by Guzzle + New Relic. In the New Relic Agent changelog they've mentioned that in version 7.3 there was some Guzzle fix, but even using the 8.0 didn't work, so there is still something wrong. In our case this was happening only in two of our scripts that were using Guzzle. We found that there are two solutions:
Set newrelic.guzzle.enabled = false in newrelic.ini. You will lose data in the External Services tab this way, but you might not need it anyway.
Downgrade New Relic Agent to version 6.x that somehow also works
If you are reading this when they've released something newer than version 8.0, you could also try to update New Relic Agent to the latest and maybe they fixed that

In my case I had deactivated the buffering function ob_start("buffer"); in my code ;)

A possible problem is PHP 7.3 + Xdebug. Please change Xdebug 2.7.0beta1 to Xdebug 2.7.0rc1 or the latest version of Xdebug.

For some reason, when I remove profile from my xdebug.ini modes, it fixes it for me.
i.e. change
xdebug.mode=debug,develop,profile
to
xdebug.mode=debug,develop

Related

Apache exit signal Segmentation fault (11) while doing POST with more than 85 fields

I had lot of exit signal Segmentation fault (11) errors in the apache log file.
After a few hours of disabling things and doing some tests I found that this happens only when I've a HTML form with more than 85 fields doing a POST over HTTPS, I mean.. if I try to do the POST with 84 fields, it works but if I try with 86 I have an error in the browser and the segmentation fault in the log file.
I tried to disable the firewall, change PHP settings and nothing works (is not related to the maximum of post fields in the php.ini), but when I tried to do that over HTTP it works fine.
I'm doing this in a simple HTML file so I think is not an issue related to PHP.
Any ideas? Suggestion? Thanks!
Apache Version
Server version: Apache/2.4.39 (Unix)
Server built: May 10 2019 11:45:46
OS
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1160.15.2.el7.x86_64
Architecture: x86-64

Error 500 after php upgrade to 7.2(Wordpress/Virtualmin)

I let my host upgrade php version to 7.2 and after that I am getting error 500 message(Internal Server Error) on my wordpress page.
I am using Virtualmin.
When opening error log I get the following:
[Mon Jan 06 16:04:41 2020] [error] (13)Permission denied: exec of '/etc/httpd/bin/suexec' failed
[Mon Jan 06 16:04:41 2020] [error] Premature end of script headers: php7.2.cgi
I searched for the solution elsewhere but didn't find any solution.
How can I solve this?
You can go to >>> virtualmin >>> serve configuration >> website options
And make the change to
FCGI, fpm or what your system is available
I can't figure out the exact solution, but the web host told me that because this server had somehow been accessed by malware, this whole thing had to be closed down and the domain moved to a new address.
Moving domain to a new server solved this problem.
This would be something to ask your Hosting Provider, Potentially they have changed the PHP Handler (Eg from suPHP to FastCGI)
you have upgraded PHP, but your Virtualmin is not configured to use it correctly.
On Virtualmin go to system settings -> Re-check configuration, Virtualmin will detect new PHP version and configure it to use on your server.

Apache + php 7 + FPM = sudden system breaks

I've an apache2 + php7 server running a simple wordpress blog.
The blog ran all the time with apache + mod_php + php7, but recently the during some access peak the system was crashing, getting to slow and even breaking.
So I googled how to optimize the configurations and many tutorials said mod_php is slow and I should replace it for php-fpm.
I did and after the change the site was noteciable faster, but now it randomly crashes and start presenting http 500 error...
There is no obvius reason for the new crashes, no users peak or any other situation I could notice.
the apache error log is plent of:
[fastcgi:error] [pid 37179] [client 162.158.167.177:26270] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php-fcgi"
[fastcgi:error] [pid 37176] (104)Connection reset by peer: [client 103.22.200.111:25406] FastCGI: comm with server "/usr/lib/cgi-bin/php-fcgi" aborted: read failed, referer: http://www.fqn.com.br/wordpress/wp-content/plugins/jetpack/css/jetpack.css
there are really thousands of errors like this, every two seconds one error and i dont understand.
First why is apache asking a css to the fpm?
Second what "/usr/lib/cgi-bin/php-fcgi" is supposed to be? there is no file in this folder!! what is supposed to be there?
The log of php-fpm is tottaly useless, I enabled the DEBUG level of loggind and what I get is just:
DEBUG: pid 1664, fpm_pctl_perform_idle_server_maintenance(), line 379: [pool www] currently 1 active children, 2 spare children, 3 running children. Spawning rate 2
every second one status message like this and at random intervals some:
WARNING: pid 1664, fpm_children_bury(), line 252: [pool www] child 38554 exited on signal 11 (SIGSEGV) after 58.797353 seconds from start
but no stack trace or detailed error message to help me to understand. I really liked the performance of apache + fpm and didn't want to roll back to mod_php, but it's impossible to run the system for 12h without crashes in the current configuration.
the bellow link shows the php_info page of the server
https://jpst.it/11FIP
does someone have an idea?
It is a programming fault.... somewhere in the php engine
and it seems to be invoked by a program setting it's input to non-blocking or stopping too soon after starting.
(exec(), shell_exec(), proc_open() which fail)...
It seems the PHP developers are not very eager, to put it mildly, to solve the issue. It has been known to exist for YEARS (2012 is the oldest ticket I have seen)
https://bugs.php.net/bug.php?id=73056

Connection reset by peer: mod_fcgid: error reading data from FastCGI server

I am having issue on PHP where my app is trying to run a php backup file and suddenly getting HTTP Error 500 Code. I have checked the logs and this what it saying.
[Tue Aug 28 14:17:28 2012] [warn] [client x.x.x.x] (104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server, referer: http://example.com/backup/backup.php
[Tue Aug 28 14:17:28 2012] [error] [client x.x.x.x] Premature end of script headers: backup.php, referer: http://example.com/backup/backup.php
Anyone knows how to fix this? I'm really stuck in here and can't find solution in internet.
Hope anyone could share their knowledge.
Thanks.
James
I managed to solved this by adding FcgidBusyTimeout . Just in case if anyone have similar issue with me.
Here is my settings on my apache.conf:
<VirtualHost *:80>
.......
<IfModule mod_fcgid.c>
FcgidBusyTimeout 3600
</IfModule>
</VirtualHost>
I had very similar errors in the Apache2 log files:
(104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server
Premature end of script headers: phpinfo.php
After checking the wrapper scripts and Apache2 settings, I realized that /var/www/ did not have accordant permissions. Thus the FCGId Wrapper scripts could not be read at all.
ls -la /var/www
drwxrws--- 5 www-data www-data 4096 Oct 7 11:17 .
For my scenario chmod -o+rx /var/www was required of course, since the used SuExec users are not member of www-data user group - and they should not be member for security reasons of course.
if you want to install a PHP version < 5.3.0, you must replace
--enable-cgi
with:
--enable-fastcgi
in your ./configure statement, excerpt from the php.net doc:
--enable-fastcgi
If this is enabled, the CGI module will be built with support for FastCGI also. Available since PHP 4.3.0
As of PHP 5.3.0 this argument no longer exists and is enabled by --enable-cgi instead. After the compilation the ./php-cgi -v should look like this:
PHP 5.2.17 (cgi-fcgi) (built: Jul 9 2013 18:28:12)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
NOTICE THE (cgi-fcgi)
I had this issue and realized that the file cgi-bin/php-fcgi had no execution rights.
It had 644 mode while is should have 755 mode.
Setting the correct mode was impossible (probably because the file was opened or sth), so I copied that file from another domain directory where it had proper rights already set and that fixed everything.
I had the same problem with a different and simple solution.
Problem
I installed PHP 5.6 following the accepted answer to this question on Ask Ubuntu.
After using Virtualmin to switch a particular virtual server from PHP 5.5 to PHP 5.6, I received a 500 Internal Server Error and had the same entries in the apache error log:
[Tue Jul 03 16:15:22.131051 2018] [fcgid:warn] [pid 24262] (104)Connection reset by peer: [client 10.20.30.40:23700] mod_fcgid: error reading data from FastCGI server
[Tue Jul 03 16:15:22.131101 2018] [core:error] [pid 24262] [client 10.20.30.40:23700] End of script output before headers: index.php
Cause
Simple: I didn't install the php5.6-cgi packet.
Fix
Installing the packet and reloading apache solved the problem:
sudo apt-get install php5.6-cgi if you are using PHP 5.6
sudo apt-get install php5-cgi if you are using a different PHP 5 version
sudo apt-get install php7.0-cgi if you are using PHP 7
Then use service apache2 reload to apply the configuration.
The famous Moodle "replace.php" script can generate this situation too.
For me it was taking ages to run and then failed with a 500 message in the browser and also with the above error message in my apache error log file.
I followed up on #james-wise answer:
FcgidBusy is readably described in the Apache documentation. I tried this: doubled the amount of time which apache would give my script to run, by inserting the following line in /etc/apache2/mods-available/fcgid.conf
FcgidBusyTimeout 600
Then I restarted Apache and tried to run my replace.php script again.
Fortunately this time the script instance ran to completion, so for my purposes this served as a solution.
I came across this one while debugging a virtualmin/apache related error.
In my case, I am running virtualmin and had in my virtual machine's php.ini
safe_mode=On.
In my Virtual Machine's error log, I was getting the fcgi Connection reset by peer: mod_fcgid: error reading data from FastCGI server
In my main apache error log I was getting:
PHP Fatal error: Directive 'safe_mode' is no longer available in PHP in Unknown on line 0
In my case, I simply set safe_mode = Off in my php.ini and restarted apache.
stackoverflow.com/questions/18683177/where-to-start-with-deprecated-directive-safe-mode-on-line-0-in-apache-error
Not in this questions askers case but often:
What does the "premature end of script headers" error mean?
That error means that the FCGI call was exited unexpectedly.
In some cases it means that the script "backup.php" did crash.
How to fix this?
If the crash of a script was the cause, fix the script so that it does not crash. Then this error is fixed, too. To find out if and why a script crashes, you need to debug it. For example you can check the PHP error log. Errors logged to STDERR normally go into the error handler of the FCGI.
I had the same problem with long-running scripts with the error messages
"Premature end of script headers: index.php" and "Connection reset by peer: mod_fcgid: error reading data from FastCGI server" in error_log.
After hours of testing this helps for me (CentOS 6, PHP-FPM 7, Plesk 12.5.30):
edit the config file:
/etc/httpd/conf.d/fcgid.conf
Set a higher running time. In my case 600 seconds
create the new entry:
FcgidBusyTimeout 600
adapt following entries:
FcgidIOTimeout 600
FcgidConnectTimeout 600
restart httpd:
service httpd restart
In CentOS releases suexec is compiled to run only in /var/www. If you try to set a DocumentRoot somewhere else you have to recompile it - the error in apache log are:
(104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server
Premature end of script headers: php5.fcgi
Just install php5-cgi
in debian
sudo apt-get install php5-cgi
in Centos
sudo yum install php5-cgi
Check /var/lib/php/session and its permissions. This dir should be writable by user so the session can be stored
As already mentioned this could be happening due to fcgi handler permission issues. If you're using suexec - don't forget to check if apache has this module enabled.
I increased max execution time to 600 seconds job done !
If you're on a shared server like me the host said it was a result of hitting memory limits, so they kill scripts which results in the "Premature end of script headers" seen in this error. They referred me to this:
https://help.dreamhost.com/hc/en-us/articles/216540488-Why-did-procwatch-kill-processes-on-my-Shared-serv
Given an increase in memory, the issues went. I think a backup plugin Updraft on wordpress was perhaps over zealous in its duty/settings.
In my case I was using a custom extension for my PHP files and I had to edit /etc/apache2/conf-available/php7.2-fpm.conf and add the following code:
<FilesMatch ".+\.YOUR_CUSTOM_EXTENSION$">
SetHandler "proxy:unix:/run/php/php7.2-fpm.sock|fcgi://localhost"
</FilesMatch>
I've tried the majority of answers that I've found on this issue. My issue was with wp-cron.php executing a particular function.
I'm working on Plesk CentOS7, Apache server.
I used this related question and suggested answer to help me find out how to adjust the fcgid.conf memory limit utilizing the command line.
Upon trying to troubleshoot the limits on the fcgid.conf file (/etc/httpd/conf.d/fcgid.conf) and restarting apache gracefully, I found that there was no change.
After seeing that other cron jobs were running properly, I decided to refer back to my function declared in functions.php and found that some of the arguments declared were not being specified properly, so there was a loop occurring that would eventually lead to the timeout.
Upon fixing this and running the cron again, it ran as it should.
Hope this helps someone else in a similar position!
I got the same problem (with Plesk 12 installed).
However, when i switched from execute PHP as FastCGI to Apache Module the website worked.
Checked my suexec log:
$ cd /var/log/apache2/
$ less suexec.log
When you find something like this:
[2015-03-22 10:49:00]: directory is writable by others: (/var/www/cgi-bin/cgi_wrapper)
[2015-03-22 10:49:05]: uid: (10004/gb) gid: (1005/1005) cmd: cgi_wrapper
try this commands
$ chown root:root /var/www/cgi-bin/cgi_wrapper
$ chmod 755 /var/www/cgi-bin/cgi_wrapper
$ shutdown -r now
as root.
I hope it can help you.

Is it posible to use symfony 2.0 with WAMP server reliably?

This is my setup:
Windows XP SP2
WAMP 2.2 (php 5.3.9, apache 2.2.21, mysql 5.5.20)
Symfony 2.0
I run into a problem which is basically this:
Symfony-2.0 vendors Apache
Everything seems to be fine:
Running mysqltest.php connects OK to the database.
localhost/web/config.php is OK (just recommends setting up intl and APC)
"php app/bin doctrine:schema:create" creates the schema successfully.
But, when I try to:
"localhost/web/app.php"
"localhost/web/app_dev.php"
Apache crashes (windows popup saying something about php5ts.dll) and the only meaningful thing on the log says:
[Fri Apr 27 05:03:54 2012] [notice] Child 3528: Starting thread to listen on port 80.
[Fri Apr 27 05:07:00 2012] [notice] Parent: child process exited with status 3221225477 -- Restarting.
[Fri Apr 27 05:07:01 2012] [notice] Apache/2.2.21 (Win32) PHP/5.3.9 configured -- resuming normal operations
The solution to the question I referenced previously is to use php 5.4, well, as stated in
How do I get PDO to work on WAMP with PHP 5.4?
besides 5.4 not being officially supported by WAMP, many extensions don't work on 5.4 (APC, PDO maybe?).
Older versions of php are also ruled out. Official WAMP addons exist only for php 5.3.0 and 5.3.1, while symfony2 requires at least 5.3.2.
I also tried this on a Windows Server 2003 machine, with the same results.
This is taking so much time I'm even considering throwing in a Linux VM on the server with my usual symfony setup.
So I need to know whether it's posible to have a stable symfony2/WAMP2.2 environment which does not imply losing extensions and symfony2 functionality such as doctrine.
Any ideas are welcome.
I could not manage to make wampserver work.
I have also tried XAMPP 1.7.7 (php 5.3.8) and other distributions as well and ran into the same issue over and over.
Apparently the error message I have been receiveing is not useful at all, as suggested here, it's just a "something crashed" message.
I couldn't find any other message out of the logs in any of the installations, so I can't tell wether this is a symfony2 bug, a php bug or an apache bug.
Anyway, the error message I got being completely useless, all I could do is moving away from Windows. I setup a LAMP(Linux) environment in virtual machine and run it from a Windos host, it's been some months and I haven't had any issues so far.
I added the following lines at the end of my 'httpd.conf' file of the running Apache server based on the explanation given here:
<IfModule mpm_winnt_module>
ThreadStackSize 8888888
</IfModule>
And so far didn't get any issues.

Categories