I have a server running LEMP stack that hosts a wide range websites. During the night, all the sites got shutdown and the message "502 bad gateway" Is displayed. I followed the stream of errors and concluded that php7.4-fpm was the issue. I need help to figure out how to solve the error below.
NOTE: During the night, no updates or changes has been made to the system
● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-07-27 09:35:38 CEST; 7s ago
Docs: man:php-fpm7.4(8)
Process: 1561620 ExecStart=/usr/sbin/php-fpm7.4 --nodaemonize --fpm-config /etc/php/7.4/fpm/php-fpm.conf (code=exited, status=78)
Process: 1561621 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
Main PID: 1561620 (code=exited, status=78)
Jul 27 09:35:38 Ubuntu-2010-groovy-64-minimal systemd[1]: Starting The PHP 7.4 FastCGI Process Manager...
Jul 27 09:35:38 Ubuntu-2010-groovy-64-minimal php-fpm7.4[1561620]: [27-Jul-2021 09:35:38] ERROR: [/etc/php/7.4/fpm/php-fpm.conf:98] value is NULL for a ZEND_INI_PARSER_ENTRY
Jul 27 09:35:38 Ubuntu-2010-groovy-64-minimal php-fpm7.4[1561620]: [27-Jul-2021 09:35:38] ERROR: failed to load configuration file '/etc/php/7.4/fpm/php-fpm.conf'
Jul 27 09:35:38 Ubuntu-2010-groovy-64-minimal php-fpm7.4[1561620]: [27-Jul-2021 09:35:38] ERROR: FPM initialization failed
Jul 27 09:35:38 Ubuntu-2010-groovy-64-minimal systemd[1]: php7.4-fpm.service: Main process exited, code=exited, status=78/CONFIG
Jul 27 09:35:38 Ubuntu-2010-groovy-64-minimal systemd[1]: php7.4-fpm.service: Failed with result 'exit-code'.
Jul 27 09:35:38 Ubuntu-2010-groovy-64-minimal systemd[1]: Failed to start The PHP 7.4 FastCGI Process Manager.
I'm grateful for any ideas that you could have!
Just an idea, as I had the same error today setting up a fresh Alma linux server with httpd and php74-php-fpm from the remi repository.
I suggest the root cause of your problem was that the config had been altered some time before your shutdown, and that bad config then lurked causing no immediate problem. At the time of the problem your servers were restarted for some reason, maybe an overnight job run by someone else, a regular update of system packages leading to daemons being restarted, or a power failure. When they tried to start up again, they used the current config, which was by that time corrupt.
In my case I was editing the FPM config file /etc/opt/remi/php74/php-fpm.d/www.conf and accidentally used a # character for a comment, e.g. # My changes. On restarting the server, I got the same value is NULL for a ZEND_INI_PARSER_ENTRY error message in the journal. The line numbers mentioned did not help, they basically referred to how php-fpm.conf includes every file in php-fpm.d which happens to include www.conf but didn't specify which line had the error.
The fix in my case was to use the correct ; (semicolon) character, and not a #, for comments in www.conf.
This may not have been your case but I hope it helps someone :)
Update: I solved this issue by removing php7.4-fpm and installing it again. However, the problem still remains. Why did php7.4-fpm suddenly stopped working?
https://www.php.net/manual/en/features.commandline.webserver.php
From 7.4+ onwards, I assume PHP built in server is capable of handling multiple incoming requests, up to and equal to the environment variable: PHP_CLI_SERVER_WORKERS
I have a web app which is composed of a couple dozen AJAX powered lists, on the first page load, using the built-in server it slows to a crawl, usually fails due to timeout in PHP scripts.
I read the above feature, added environment variable (PHP is a docker container), and shell'ed into my container, did a top/ps I can now see X number of PHP processes:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 722864 21932 14448 S 0.3 1.1 0:09.91 symfony
20 root 20 0 210892 48188 36544 S 0.0 2.4 0:00.79 php7.4
21 root 20 0 205676 33460 24500 S 0.0 1.6 0:00.22 php7.4
22 root 20 0 208212 40908 29640 S 0.0 2.0 0:00.42 php7.4
23 root 20 0 210644 42236 30836 S 0.0 2.1 0:00.61 php7.4
24 root 20 0 208764 40784 31176 S 0.0 2.0 0:01.14 php7.4
25 root 20 0 205804 33588 24508 S 0.0 1.6 0:00.22 php7.4
...
I am using Symfony to start a dev server, but no matter what I do none of the processes seem to be carrying any of the load? What am I missing?
I switched to the built-in PHP server about 2 years ago, from NGINX. It made my vagrant setup easier (now docker) but my performance took a hit, which I've dealt with. I'd like to improve the responsiveness of the app using this approach if possible.
Any ideas?
We are having problems with our containers in our ECS service. The purpose of this service is to receive data and save it to s3. Service is running fine until it is being used in production and healtcheck start failing.
For our healthcheck we use this line to determine php-fpm state:
cgi-fcgi -bind -connect localhost:9001
From tailing php fpm log in /var/log I can see this line over and over again:
[pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers)
Now I found many articles explaining how to set php fpm settings for my application, primarily this one: https://thisinterestsme.com/php-fpm-settings/
Based on the article I can see that we have 2 cores and each php-fpm process averages 40MB. Our total RAM is around 16GB. Based on these settings I can calculate php-fpm server values:
pm.max_children = 16GB / 40MB = 400
pm.start_servers = 2 cores x 4 = 8
pm.max_spare_servers = 2 cores x 4 = 8
pm.min_spare_servers = 2 cores x 2 = 4
I tried way bigger settings as well like start_servers = 32, but I am still receiving same warnings, which makes me think something is not configured properly.
I've an apache2 + php7 server running a simple wordpress blog.
The blog ran all the time with apache + mod_php + php7, but recently the during some access peak the system was crashing, getting to slow and even breaking.
So I googled how to optimize the configurations and many tutorials said mod_php is slow and I should replace it for php-fpm.
I did and after the change the site was noteciable faster, but now it randomly crashes and start presenting http 500 error...
There is no obvius reason for the new crashes, no users peak or any other situation I could notice.
the apache error log is plent of:
[fastcgi:error] [pid 37179] [client 162.158.167.177:26270] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php-fcgi"
[fastcgi:error] [pid 37176] (104)Connection reset by peer: [client 103.22.200.111:25406] FastCGI: comm with server "/usr/lib/cgi-bin/php-fcgi" aborted: read failed, referer: http://www.fqn.com.br/wordpress/wp-content/plugins/jetpack/css/jetpack.css
there are really thousands of errors like this, every two seconds one error and i dont understand.
First why is apache asking a css to the fpm?
Second what "/usr/lib/cgi-bin/php-fcgi" is supposed to be? there is no file in this folder!! what is supposed to be there?
The log of php-fpm is tottaly useless, I enabled the DEBUG level of loggind and what I get is just:
DEBUG: pid 1664, fpm_pctl_perform_idle_server_maintenance(), line 379: [pool www] currently 1 active children, 2 spare children, 3 running children. Spawning rate 2
every second one status message like this and at random intervals some:
WARNING: pid 1664, fpm_children_bury(), line 252: [pool www] child 38554 exited on signal 11 (SIGSEGV) after 58.797353 seconds from start
but no stack trace or detailed error message to help me to understand. I really liked the performance of apache + fpm and didn't want to roll back to mod_php, but it's impossible to run the system for 12h without crashes in the current configuration.
the bellow link shows the php_info page of the server
https://jpst.it/11FIP
does someone have an idea?
It is a programming fault.... somewhere in the php engine
and it seems to be invoked by a program setting it's input to non-blocking or stopping too soon after starting.
(exec(), shell_exec(), proc_open() which fail)...
It seems the PHP developers are not very eager, to put it mildly, to solve the issue. It has been known to exist for YEARS (2012 is the oldest ticket I have seen)
https://bugs.php.net/bug.php?id=73056
Our application runs in a Docker container on AWS:
Operating system: Ubuntu 14.04.2 LTS (Trusty Tahr)
Nginx version: nginx/1.4.6 (Ubuntu)
Memcached version: memcached 1.4.14
PHP version: PHP 5.5.9-1ubuntu4.11 (cli) (built: Jul 2 2015 15:23:08)
System Memory: 7.5 GB
We get blank pages and a 404 Error less frequently. While checking the logs, I found that the php-child process is killed and it seems that memory is mostly used by memcache and php-fpm process and very low free memory.
memcache is configured to use 2 GB memory.
Here is php www.conf
pm = dynamic
pm.max_children = 30
pm.start_servers = 9
pm.min_spare_servers = 4
pm.max_spare_servers = 14
rlimit_files = 131072
rlimit_core = unlimited
Error logs
/var/log/nginx/php5-fpm.log
[29-Jul-2015 14:37:09] WARNING: [pool www] child 259 exited on signal 11 (SIGSEGV - core dumped) after 1339.412219 seconds from start
/var/log/nginx/error.log
2015/07/29 14:37:09 [error] 141#0: *2810 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: x.x.x.x, server: _, request: "GET /suggestions/business?q=Selectfrom HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "http://example.com/"
/var/log/nginx/php5-fpm.log
[29-Jul-2015 14:37:09] NOTICE: [pool www] child 375 started
/var/log/nginx/php5-fpm.log:[29-Jul-2015 14:37:56] WARNING: [pool www] child 290 exited on signal 11 (SIGSEGV - core dumped) after 1078.606356 seconds from start
Coredump
Core was generated by php-fpm: pool www.Program terminated with signal SIGSEGV, Segmentation fault.#0 0x00007f41ccaea13a in memcached_io_readline(memcached_server_st*, char*, unsigned long, unsigned long&) () from /usr/lib/x86_64-linux-gnu/libmemcached.so.10
dmesg
[Wed Jul 29 14:26:15 2015] php5-fpm[12193]: segfault at 7f41c9e8e2da ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:28:26 2015] php5-fpm[12211]: segfault at 7f41c966b2da ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:29:16 2015] php5-fpm[12371]: segfault at 7f41c9e972da ip 00007f41ccaea13a sp 00007ffcc5730b70 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:35:36 2015] php5-fpm[12469]: segfault at 7f41c96961e9 ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:35:43 2015] php5-fpm[12142]: segfault at 7f41c9e6c2bd ip 00007f41ccaea13a sp 00007ffcc5730b70 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:37:07 2015] php5-fpm[11917]: segfault at 7f41c9dd22bd ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:37:54 2015] php5-fpm[12083]: segfault at 7f41c9db72bd ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
While googling for this same issue, and trying hard to find a solution that was not related to sessions (because I have ruled that out) nor to bad PHP code (because I have several websites running precisely the same version of WordPress, and none have issues... except for one), I came upon an answer telling that a possible solution did involve removing some buggy extension (usually memcache/d, but could be something else).
Since I had this same site working flawlessly on one Ubuntu server, when switching to a newer server, I immediately suspected that it was the migration from PHP 5.5 to 7 that caused the problem. It was just strange because no other website was affected. Then I remembered that another thing was different on this new server: I had also installed New Relic. This is both an extension and a small server that runs in the background and sends a lot of analytics data to New Relic for processing. Allegedly, it's a PHP 5 extension, but, surprisingly, it loads well on PHP 7, too.
Now here comes the tricky bit. At some point, I had installed W3 Total Cache for the WordPress installation of that particular website. Subsequently, I saw that the performance of that server was so stellar that W3TC was unnecessary, and simply stuck to a much simpler configuration. So I could uninstall W3TC. That's all very nice, but... I forgot that I had turned New Relic on W3TC, too (allegedly, it adds some extra analytics data to be sent to New Relic). When uninstalling W3TC, probably there was 'something' left on the New Relic configuration in my server which was still attempting to send data through the W3TC interface (assuming that W3TC has an interface... I really have no idea how it works at that level), and, because that specific bit of code was missing, the php_fpm handler for that website would fail... some of the time. Not all the time, because I'm assuming that, in most cases, nginx was sending static pages back. Or maybe php_fpm, set to 'recycle' after 100 calls or so, would crash-on-stop. Whatever exactly was happening, it was definitely related to New Relic — as soon as I removed the New Relic extension from PHP, that website went back to working normally.
Because this is such a specific scenario, I'm just writing this as an answer, in the remote chance that someone in the future googles for the exact problem.
In my case it was related to zend debug/xdebug. It forwards some TCP packets to the IDE (PhpStorm), that was not listening on this port (debugging was off). The solution is to either disable these extensions or enable debug listening on the debugging port.
I had this problem after installing xdebug, adding some properties to /etc/php/7.1/fpm/php.ini and restarting nginx. This is running on a Homestead Laravel box.
Simply restarting the php7.1-fpm service solved it for me.
It can happen if PHP is unable to write the session information to a file. By default it is /var/lib/php/session. You can change it by using configuration session_save_path.
phpMyAdmin having problems on nginx and php-fpm on RHEL 6
In my case it was Xdebug. After uninstalling it, it got back to normal.
In my case, it was caused by the New Relic PHP Agent. Therefore, for a specific function that caused a crash, I added this code to disable New Relic:
if (function_exists('newrelic_ignore_transaction')) {
newrelic_ignore_transaction();
}
Refer to: https://discuss.newrelic.com/t/how-to-disable-a-specific-transaction-in-php-agent/42384/2
In our case it was caused by Guzzle + New Relic. In the New Relic Agent changelog they've mentioned that in version 7.3 there was some Guzzle fix, but even using the 8.0 didn't work, so there is still something wrong. In our case this was happening only in two of our scripts that were using Guzzle. We found that there are two solutions:
Set newrelic.guzzle.enabled = false in newrelic.ini. You will lose data in the External Services tab this way, but you might not need it anyway.
Downgrade New Relic Agent to version 6.x that somehow also works
If you are reading this when they've released something newer than version 8.0, you could also try to update New Relic Agent to the latest and maybe they fixed that
In my case I had deactivated the buffering function ob_start("buffer"); in my code ;)
A possible problem is PHP 7.3 + Xdebug. Please change Xdebug 2.7.0beta1 to Xdebug 2.7.0rc1 or the latest version of Xdebug.
For some reason, when I remove profile from my xdebug.ini modes, it fixes it for me.
i.e. change
xdebug.mode=debug,develop,profile
to
xdebug.mode=debug,develop