Nginx error recv() failed (104: Connection reset by peer) - php

Question

Since a couple of days ago, I'm getting some errors on my server. I use CentOS 6.5 with Parallels 12.0.18, Apache server to serve dynamic content and Nginx as proxy to serve static content.
At first, I was getting the following error:
[error] 29951#0: *5138862 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 89.7.24.108, server: , request: "GET /page/2/ HTTP/1.1", upstream: "http://ip:7080/page/2/", host: "domain.es", referrer: "http://domain.es/"
Then, I changed some configuration, like increasing MaxClients on my "httpd.conf" file and this lines to my /etc/nginx/conf.d/timeout.conf file:
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
It all seemed to be working fine until I got the same errors again, along with a new one:
[error] 15228#0: *130292 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 89.130.25.154, server: domain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:7080/", host: "domain.com"
I have two different websites on the same server. That's why you see two different hosts in there.
Here's the problem: when I got these errors, I got a "502 Bad Gateway" and the server becomes so slow that I can't even log in using the SSH terminal. I can only fix it temporally by resetting the httpd service.
I know there are other topics similar to this one, but all I found were problems with PHP-FPM, which I don't use.
Here's my Nginx configuration file:
user nginx;
worker_processes 16;
error_log /var/log/nginx/error.log;
events {
worker_connections 1024;
}
http {
server_names_hash_max_size 2048;
server_names_hash_bucket_size 512;
server_tokens off;
include mime.types;
default_type application/octet-stream;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 10;
# Gzip on
gzip on;
gzip_vary on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_buffers 4 32k;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;
gzip_disable "MSIE [1-6]\.";
# Other configurations
ignore_invalid_headers on;
client_max_body_size 8m;
client_header_timeout 3m;
client_body_timeout 3m;
#send_timeout 3m;
connection_pool_size 256;
client_header_buffer_size 4k;
large_client_header_buffers 4 32k;
request_pool_size 4k;
output_buffers 4 32k;
postpone_output 1460;
# Cache most accessed static files
open_file_cache max=10000 inactive=10m;
open_file_cache_valid 2m;
open_file_cache_min_uses 1;
open_file_cache_errors on;
# virtual hosts includes
include "/etc/nginx/conf.d/*.conf";
}
Here's my Nginx vhost file:
server {
listen ip:80 default_server;
server_name domain.es;
server_name www.domain.es;
server_name ipv4.domain.es;
client_max_body_size 128m;
root "/var/www/vhosts/domain.es/httpdocs";
access_log "/var/www/vhosts/system/domain.es/logs/proxy_access_log";
error_log "/var/www/vhosts/system/domain.es/logs/proxy_error_log";
if ($host ~* ^www.domain.es$) {
rewrite ^(.*)$ http://domain.es$1 permanent;
}
location / {
proxy_pass http://82.194.74.41:7080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
access_log off;
}
location #fallback {
proxy_pass http://ip:7080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
access_log off;
}
location ~ ^/plesk-stat/ {
proxy_pass http://ip:7080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
access_log off;
}
location ~ ^/(.*\.(ac3|avi|bmp|bz2|css|cue|dat|doc|docx|dts|exe|flv|gif|gz|htm|html|ico|img|iso|jpeg|jpg|js|mkv|mp3|mp4|mpeg|mpg|ogg|pdf|png|ppt|pptx|qt|rar|rm|swf|tar|tgz|txt|wav|xls|xlsx|zip))$ {
access_log off;
expires 7d;
add_header Cache-Control public;
try_files $uri #fallback;
}
include "/var/www/vhosts/system/domain.es/conf/vhost_nginx.conf";
}
And some of the configuration vars I use with Apache (httpd.conf):
<IfModule prefork.c>
StartServers 14
MinSpareServers 8
MaxSpareServers 14
ServerLimit 1000
MaxClients 1000
MaxRequestsPerChild 2000
</IfModule>
Thank you very much in advance!

Answer 1

In my case, a php extension is missing, after I turned it off, it recovered! Check /var/log/messages to see if there is any segfault.

Answer 2

It seems your Apache is more busy than your Nginx. When Nginx get some requests but Apache can't handle, you get '502 Bad Gateway', which meas Apache refused to work for Nginx.
Try decrease 'worker_connections' and 'worker_processes' in Nginx and increase 'MaxClients', 'ServerLimit'
Make sure
worker_connections * worker_processes < MaxClients < ServerLimit

Answer 3

In a case I saw recently this error was being logged for 95% of requests but both the web server and the apps server were almost idle with only a few connections open. The other 5% were successful.
It turned out that our friendly networking colleagues had configured an IPS device between the web and apps server and it started clawing back and dropping connections when traffic increased because it thought there was a brute force attack happening. A false-positive. So nothing to do with Nginx or Apache, but reconfiguring the IPS fixed the issue.

Answer 4

Trace your code to detect if you have any calls like "opendir" without "closedir".