I am getting a 502 Bad Gateway from Nginx on a line of PHP code that is working fine in other places of my program ($this->provider = new OAuthProvider();), and that have worked fine before. This is the message I get in the Nginx error log for each 502:
recv() failed (104: Connection reset by peer) while reading response header from upstream
In the PHP-FPM log there is a warning for each 502:
[WARNING] [pool www] child 17427 exited on signal 11 SIGSEGV after 142070.657176 seconds from start
After trying a number of changes to the nginx.conf I am stuck and would very much appreciate any pointers of what to do next.
I'm running Nginx 0.7.67 and PHP 5.3.2 on Ubuntu 10.04.
maybe http://pecl.php.net/bugs/bug.php?id=17689 or bug id #18138
Your PHP process crashed with a segfault ("signal 11 SIGSEGV"), which caused Nginx to see "connection reset by peer" (PHP is the "peer" in this case, and Nginx is telling you "Look, he hung up on me before I could get an answer from him").
Check out the PHP Bug database page on how to report a bug someone will want to fix to find out how to get a backtrace of the segfault so you can report it.
i had the same problem with APC. so i removed it and installed eaccelerator instead. no problem so far.
I had similar problems with nginx/lighttpd + php-fcgi(using spawn-fcgi), do you use any opcode cache for php?
What i found quite some time ago is that xcache was causing strange behaviour in php-fcgi, some php-fcgi processes randomly died, i was unable to find any pattern. I would recommend to take a look at apc(or other opcode cache) settings, if you are using any.
Right now im using nginx + php-fpm on freebsd and have no problems.
ZendOptimizer + APC + php-fpm 5.2.14 gives constantly reproducible SIGSEGV even on phpinfo();.
Try to switch suhosin off. Sometimes it crashes Apache.
Related
I've an apache2 + php7 server running a simple wordpress blog.
The blog ran all the time with apache + mod_php + php7, but recently the during some access peak the system was crashing, getting to slow and even breaking.
So I googled how to optimize the configurations and many tutorials said mod_php is slow and I should replace it for php-fpm.
I did and after the change the site was noteciable faster, but now it randomly crashes and start presenting http 500 error...
There is no obvius reason for the new crashes, no users peak or any other situation I could notice.
the apache error log is plent of:
[fastcgi:error] [pid 37179] [client 162.158.167.177:26270] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php-fcgi"
[fastcgi:error] [pid 37176] (104)Connection reset by peer: [client 103.22.200.111:25406] FastCGI: comm with server "/usr/lib/cgi-bin/php-fcgi" aborted: read failed, referer: http://www.fqn.com.br/wordpress/wp-content/plugins/jetpack/css/jetpack.css
there are really thousands of errors like this, every two seconds one error and i dont understand.
First why is apache asking a css to the fpm?
Second what "/usr/lib/cgi-bin/php-fcgi" is supposed to be? there is no file in this folder!! what is supposed to be there?
The log of php-fpm is tottaly useless, I enabled the DEBUG level of loggind and what I get is just:
DEBUG: pid 1664, fpm_pctl_perform_idle_server_maintenance(), line 379: [pool www] currently 1 active children, 2 spare children, 3 running children. Spawning rate 2
every second one status message like this and at random intervals some:
WARNING: pid 1664, fpm_children_bury(), line 252: [pool www] child 38554 exited on signal 11 (SIGSEGV) after 58.797353 seconds from start
but no stack trace or detailed error message to help me to understand. I really liked the performance of apache + fpm and didn't want to roll back to mod_php, but it's impossible to run the system for 12h without crashes in the current configuration.
the bellow link shows the php_info page of the server
https://jpst.it/11FIP
does someone have an idea?
It is a programming fault.... somewhere in the php engine
and it seems to be invoked by a program setting it's input to non-blocking or stopping too soon after starting.
(exec(), shell_exec(), proc_open() which fail)...
It seems the PHP developers are not very eager, to put it mildly, to solve the issue. It has been known to exist for YEARS (2012 is the oldest ticket I have seen)
https://bugs.php.net/bug.php?id=73056
Today I meet a strange question:
our project need to online a new module
when I execute artisan down in online environment and visit the site, I found it isn't show the maintenance page, it also show the home page
I check the CheckForMantenance Middleware has aleady add to global route middleware, the down file in storage/framework is exist
I execute php index.php it return the maintenance page, but When I visit the site from brown or curl it show the home page
I also run in test server and local, it all work well
I add a new route for test middlware, and visit the url used by curl and brown, and result is 404, the route does not found
I think it may be caused by router cache, but there is no cache file on bootstrap/cache or storage/framework, because I never open router cache!
I have no idea, so I modify the index file, add write header function at top, and redirect to a error html, the crazy thing happened it also show the home page!!!!!!
What happend? I'm sure the project path is right
Finally I reload the php-fpm and it recovery normal, the maintenance view work, redirect url work, and route normal
I don't understand still now, but I guess it may by caused by opcache?
I open the opcache ext, and use the default setting;
env:
laravel: 5.3
nginx: 1.8.1
php-fpm: 7.0.9 with opcache ext
First check your fpm logs, usually, something like this will pop up (check debug/log levels) between the notices:
[01-Mar-2017 23:59:45] NOTICE: [pool www] child 16951 started
[01-Mar-2017 23:59:48] WARNING: [pool www] child 14754
exited on signal 11 (SIGSEGV - core dumped) after 4393.427133 seconds
from start
You have to disable opcache unfortunately. I've been seeing this issue since php 5.5 all the way to 7.1 , you will also find these in the error logs:
2017/03/02 10:00:24 [error] 30498#30498: *170523 upstream timed out
(110: Connection timed out) while reading response header from upstream,
client: 81.243.144.1xx, server: fake.test.pro,
request: "POST /api/users/53e4203cfd1c46e08d5b570c2c93ff86/items HTTP/1.1",
upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock", host: "fake.test.pro",
referrer: "http://fake.test.pro/console"
In particular with Laravel, but I've also seen it on wordpress installations. It stops when I disable opcache on all versions of php-fpm
There are bugreports around on this issue but no fixes so far. I always end up doing this :
[opcache]
; Determines if Zend OPCache is enabled
opcache.enable=0
in /etc/php/7.*/fpm/php.ini files. Then my application is robust again and it costs us 150ms. it sucks.
Problem
When xdebug server is running from IntelliJ IDEA, I get 502 Bad Gateway from nginx when I try loading my site to trigger breakpoints.
If I stop the xdebug server, the site works as intended.
So, I'm not able to run the debugger, but it did work previously (!). Not able to pinpoint why it suddenly stopped working.
Setup
A short explanation of the setup (let me know if I need to expand on this).
My php app is running in a docker container, and it is linked to nginx running in a different container using volumes_fromin the docker compose config.
After starting the app, I can verify using phpinfo(); the xdebug module is loaded.
My xdebug.ini has the following content:
zend_extension=xdebug.so
xdebug.remote_enable=1
xdebug.remote_host=10.0.2.2
xdebug.remote_connect_back=0
xdebug.remote_port=5555
xdebug.idekey=complex
xdebug.remote_handler=dbgp
xdebug.remote_log=/var/log/xdebug.log
xdebug.remote_autostart=1
I got the ip address for remote_host (where the xdebug server is running) by these steps:
docker-machine ssh default
route -n | awk '/UG[ \t]/{print $2}' <-- Returns 10.0.2.2
To verify I could reach the debugging server from within my php container, I did the following steps
docker exec -it randomhash bash
nc -z -v 10.0.2.2 5555
Giving the following output depending on xdebug server running or not:
Running: Connection to 10.0.2.2 5555 port [tcp/*] succeeded!
Not running: nc: connect to 10.0.2.2 port 5555 (tcp) failed: Connection refused
So IntelliJ IDEA is surely set up to receive connections on 5555. I also did the appropriate path mapping between my source file paths and the remote path (when setting up the PHP Remote Debugging server from within IDEA).
Any ideas? Kind of lost on this one as I don't have much experience with any of these technologies :D
This sometimes happens, the reason is the errors in php-fpm and xdebug (exactly)!
When I refactored my colleagues code, оne page on the project returned 502 Bad Gateway
Here's what I found:
php-fpm.log
WARNING: [pool www] child 158 said into stderr: "*** Error in `php-fpm: pool www': free(): invalid size: 0x00007f1351b7d2a0 ***"
........
........
WARNING: [pool www] child 158 exited on signal 6 (SIGABRT - core dumped) after 38.407847 seconds from start
I found a piece of code that caused the error:
ob_start();
$result = eval("?>".$string."<"."?p"."hp return 1;");
$new_string = ob_get_clean();
But that is not all. The error occurred only in a certain state $string which at first glance, did not differ from the others. In my case, everything is simple. I removed the code that caused the error. This did not affect the functionality of the web page. I continued to debug the code further.
I had the same problem with the Vagrant Homestead Parallels box with a Silicon chip. Switching from php 7.3 to 7.4 fixed the issue for me.
Our application runs in a Docker container on AWS:
Operating system: Ubuntu 14.04.2 LTS (Trusty Tahr)
Nginx version: nginx/1.4.6 (Ubuntu)
Memcached version: memcached 1.4.14
PHP version: PHP 5.5.9-1ubuntu4.11 (cli) (built: Jul 2 2015 15:23:08)
System Memory: 7.5 GB
We get blank pages and a 404 Error less frequently. While checking the logs, I found that the php-child process is killed and it seems that memory is mostly used by memcache and php-fpm process and very low free memory.
memcache is configured to use 2 GB memory.
Here is php www.conf
pm = dynamic
pm.max_children = 30
pm.start_servers = 9
pm.min_spare_servers = 4
pm.max_spare_servers = 14
rlimit_files = 131072
rlimit_core = unlimited
Error logs
/var/log/nginx/php5-fpm.log
[29-Jul-2015 14:37:09] WARNING: [pool www] child 259 exited on signal 11 (SIGSEGV - core dumped) after 1339.412219 seconds from start
/var/log/nginx/error.log
2015/07/29 14:37:09 [error] 141#0: *2810 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: x.x.x.x, server: _, request: "GET /suggestions/business?q=Selectfrom HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "http://example.com/"
/var/log/nginx/php5-fpm.log
[29-Jul-2015 14:37:09] NOTICE: [pool www] child 375 started
/var/log/nginx/php5-fpm.log:[29-Jul-2015 14:37:56] WARNING: [pool www] child 290 exited on signal 11 (SIGSEGV - core dumped) after 1078.606356 seconds from start
Coredump
Core was generated by php-fpm: pool www.Program terminated with signal SIGSEGV, Segmentation fault.#0 0x00007f41ccaea13a in memcached_io_readline(memcached_server_st*, char*, unsigned long, unsigned long&) () from /usr/lib/x86_64-linux-gnu/libmemcached.so.10
dmesg
[Wed Jul 29 14:26:15 2015] php5-fpm[12193]: segfault at 7f41c9e8e2da ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:28:26 2015] php5-fpm[12211]: segfault at 7f41c966b2da ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:29:16 2015] php5-fpm[12371]: segfault at 7f41c9e972da ip 00007f41ccaea13a sp 00007ffcc5730b70 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:35:36 2015] php5-fpm[12469]: segfault at 7f41c96961e9 ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:35:43 2015] php5-fpm[12142]: segfault at 7f41c9e6c2bd ip 00007f41ccaea13a sp 00007ffcc5730b70 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:37:07 2015] php5-fpm[11917]: segfault at 7f41c9dd22bd ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:37:54 2015] php5-fpm[12083]: segfault at 7f41c9db72bd ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
While googling for this same issue, and trying hard to find a solution that was not related to sessions (because I have ruled that out) nor to bad PHP code (because I have several websites running precisely the same version of WordPress, and none have issues... except for one), I came upon an answer telling that a possible solution did involve removing some buggy extension (usually memcache/d, but could be something else).
Since I had this same site working flawlessly on one Ubuntu server, when switching to a newer server, I immediately suspected that it was the migration from PHP 5.5 to 7 that caused the problem. It was just strange because no other website was affected. Then I remembered that another thing was different on this new server: I had also installed New Relic. This is both an extension and a small server that runs in the background and sends a lot of analytics data to New Relic for processing. Allegedly, it's a PHP 5 extension, but, surprisingly, it loads well on PHP 7, too.
Now here comes the tricky bit. At some point, I had installed W3 Total Cache for the WordPress installation of that particular website. Subsequently, I saw that the performance of that server was so stellar that W3TC was unnecessary, and simply stuck to a much simpler configuration. So I could uninstall W3TC. That's all very nice, but... I forgot that I had turned New Relic on W3TC, too (allegedly, it adds some extra analytics data to be sent to New Relic). When uninstalling W3TC, probably there was 'something' left on the New Relic configuration in my server which was still attempting to send data through the W3TC interface (assuming that W3TC has an interface... I really have no idea how it works at that level), and, because that specific bit of code was missing, the php_fpm handler for that website would fail... some of the time. Not all the time, because I'm assuming that, in most cases, nginx was sending static pages back. Or maybe php_fpm, set to 'recycle' after 100 calls or so, would crash-on-stop. Whatever exactly was happening, it was definitely related to New Relic — as soon as I removed the New Relic extension from PHP, that website went back to working normally.
Because this is such a specific scenario, I'm just writing this as an answer, in the remote chance that someone in the future googles for the exact problem.
In my case it was related to zend debug/xdebug. It forwards some TCP packets to the IDE (PhpStorm), that was not listening on this port (debugging was off). The solution is to either disable these extensions or enable debug listening on the debugging port.
I had this problem after installing xdebug, adding some properties to /etc/php/7.1/fpm/php.ini and restarting nginx. This is running on a Homestead Laravel box.
Simply restarting the php7.1-fpm service solved it for me.
It can happen if PHP is unable to write the session information to a file. By default it is /var/lib/php/session. You can change it by using configuration session_save_path.
phpMyAdmin having problems on nginx and php-fpm on RHEL 6
In my case it was Xdebug. After uninstalling it, it got back to normal.
In my case, it was caused by the New Relic PHP Agent. Therefore, for a specific function that caused a crash, I added this code to disable New Relic:
if (function_exists('newrelic_ignore_transaction')) {
newrelic_ignore_transaction();
}
Refer to: https://discuss.newrelic.com/t/how-to-disable-a-specific-transaction-in-php-agent/42384/2
In our case it was caused by Guzzle + New Relic. In the New Relic Agent changelog they've mentioned that in version 7.3 there was some Guzzle fix, but even using the 8.0 didn't work, so there is still something wrong. In our case this was happening only in two of our scripts that were using Guzzle. We found that there are two solutions:
Set newrelic.guzzle.enabled = false in newrelic.ini. You will lose data in the External Services tab this way, but you might not need it anyway.
Downgrade New Relic Agent to version 6.x that somehow also works
If you are reading this when they've released something newer than version 8.0, you could also try to update New Relic Agent to the latest and maybe they fixed that
In my case I had deactivated the buffering function ob_start("buffer"); in my code ;)
A possible problem is PHP 7.3 + Xdebug. Please change Xdebug 2.7.0beta1 to Xdebug 2.7.0rc1 or the latest version of Xdebug.
For some reason, when I remove profile from my xdebug.ini modes, it fixes it for me.
i.e. change
xdebug.mode=debug,develop,profile
to
xdebug.mode=debug,develop
I run a PHP 5.5/5.6 instance over php-fpm without any problems (unix sockets and nginx).
Today I compiled PHP 7 and also started it as a third PHP instance. While requests get served fine with the PHP 5.5/5.6 instances for the PHP 7 instance I get:
WARNING: pid 2582, fpm_request_check_timed_out(), line 277: [pool www0] child 2813, script '' (request: " ") execution timed out (120.018160 sec), terminating
in my php-fpm log file and
[error] 1889#0: *4 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.20, server: localhost, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php70-fpm.sock:", host: "xxx.xxx.xxx.21"
in my nginx error log file.
I use fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name which also works for the PHP 5.6 instance. Although the script filename for PHP 7 seems to be empty. I'm requesting a script /info.php.
Is there any difference between using PHP 5.x and 7.0 with php-fpm?
UPDATE
At least I'm not the only one who is facing those problems as it seems. I opened a bug report here: https://bugs.php.net/bug.php?id=69850
There is also a very similar (probably identical) bug already reported over here: https://bugs.php.net/bug.php?id=69875 (thx to #crypticツ)
UPDATE 2
Try the newest alpha2. The bug is fixed there. Unix sockets work now like a charm again.
I am not sure if this can be considered as answer, but at this moment, when I am writing this answer, the bug you have raised has been deemed confirmed by the PHP Team:
[2015-06-23 12:47 UTC] remi#php.net
I confirm
PHP 5.6.10 works
PHP 7.0.0-dev (Jun 17 2015)
- network socket: OK.
- UDS: broken
Update: I just saw that you are aware of the same too!