Nginx + Php Fpm scaling issues on AWS EC2 instance - php

Question

We are performing a load test using locust(1000 Users) on a webpage of our application.
Instance type: t3a.medium
The instance is running behind a load balancer. And we are using RDS Aurora Database which peaks at around 70% CPU utilization. EC2 instance metrics are healthy. EDIT: Instance memory consumption is within 800 MB out of available 4 GB
There are multiple 502 Server error: Bad Gateway and sometimes 500 and 520 errors as well.
Error 1:
2020/10/08 16:58:21 [error] 4344#4344: *41841 connect() to unix:/var/run/php/php7.2-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: <PublicIP>, server: <Domain name>, request: "GET <webpage> HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "<Domain name>"
Error 2(Alert):
2020/10/08 19:15:11 [alert] 9109#9109: *105735 socket() failed (24: Too many open files) while connecting to upstream, client: <PublicIP>, server: <Domain name>, request: "GET <webpage> HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "<Domain name>"
Listing down configuration files:
Nginx Configuration
server {
listen 80;
listen [::]:80;
root /var/www/####;
index index.php;
access_log /var/log/nginx/###access.log;
error_log /var/log/nginx/####error.log ;
server_name #####;
client_max_body_size 100M;
autoindex off;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~ \.php$ {
include fastcgi_params;
fastcgi_intercept_errors on;
fastcgi_index index.php;
fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
}
}
/etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 8096;
multi_accept on;
use epoll;
epoll_events 512;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
gzip on;
gzip_comp_level 2;
gzip_min_length 1000;
gzip_types text/xml text/css;
gzip_http_version 1.1;
gzip_vary on;
gzip_disable "MSIE [4-6] \.";
include /etc/nginx/conf.d/*.conf;
}
/etc/php/7.2/fpm/php-fpm.conf
emergency_restart_threshold 10
emergency_restart_interval 1m
process_control_timeout 10s
Php-fpm Important Parameters:
user = www-data
group = www-data
listen = /run/php/php7.2-fpm.sock
listen.owner = www-data
listen.group = www-data
;listen.mode = 0660
pm = static
pm.max_children = 300
/etc/security/limits.conf
nginx soft nofile 30000
nginx hard nofile 50000
/etc/sysctl.conf
net.nf_conntrack_max = 131072
net.core.somaxconn = 131072
net.core.netdev_max_backlog = 65535
kernel.msgmnb = 131072
kernel.msgmax = 131072
fs.file-max = 131072
What are we missing? Can anyone please point to the right direction?

Answer 1

So we were able to resolve this issue. The problem was php-fpm did not have access to access system resources. You may need to change values according to hardware specifications.
So, our final configuration looks like this:
In /etc/security/limits.conf, add following lines:
nginx soft nofile 10000
nginx hard nofile 30000
root soft nofile 10000
root hard nofile 30000
www-data soft nofile 10000
www-data hard nofile 30000
In /etc/sysctl.conf, add following values
net.nf_conntrack_max = 231072
net.core.somaxconn = 231072
net.core.netdev_max_backlog = 65535
kernel.msgmnb = 231072
kernel.msgmax = 231072
fs.file-max = 70000
In /etc/nginx/nginx.conf, change or add so finally it should have these values(kindly change them according to your use case and server capacity):
worker_processes auto;
worker_rlimit_nofile 30000;
events {
worker_connections 8096;
multi_accept on;
use epoll;
epoll_events 512;
}
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
gzip on;
gzip_comp_level 2;
gzip_min_length 1000;
gzip_types text/xml text/css;
gzip_http_version 1.1;
gzip_vary on;
gzip_disable "MSIE [4-6] .";
In /etc/php/7.2/fpm/php-fpm.conf , change values to look like this:
emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s
rlimit_files = 10000
In /etc/php/7.2/fpm/pool.d/www.conf , change values to look like this:
user = www-data
group = www-data
listen.backlog = 4096
listen.owner = www-data
listen.group = www-data
;listen.mode = 0660
pm = static
pm.max_children = 1000