Server performance issue while running cakephp web application - php

Question

Our application server is getting down or slow randomly throughout the day, CakePHP 2 application with Mysql is running on this server. We have some cronjobs set up and all are working perfectly.
This performance issue mostly occurs at business time (Day time)
Server configuration: AWS instance t2.large, FreeBSD 10.3-RELEASE-p11, Disk space 20% free (30GB)
I go through many server logs as well as application logs like below
Nginx error log (Few lines from the log)
2020/04/30 23:04:57 [info] 66440#101049: *71645 client closed connection while waiting for request, client: XX.XX.XX.XX, server: 0.0.0.0:80
2020/04/30 23:05:01 [info] 66440#101049: *71820 kevent() reported that client XX.XX.XX.XX closed keepalive connection
2020/04/30 23:05:42 [info] 66440#101049: *72494 peer closed connection in SSL handshake while SSL handshaking, client: XX.XX.XX.XX, server: 0.0.0.0:443
dmesg.today Log (Few lines)
sonewconn: pcb 0xfffff800a70cf7a8: Listen queue overflow: 193 already in queue awaiting acceptance (62 occurrences)
sonewconn: pcb 0xfffff800a70cf7a8: Listen queue overflow: 193 already in queue awaiting acceptance (57 occurrences)
sonewconn: pcb 0xfffff80115d9e7a8: Listen queue overflow: 193 already in queue awaiting acceptance (63 occurrences)
sonewconn: pcb 0xfffff80115d9e7a8: Listen queue overflow: 193 already in queue awaiting acceptance (126 occurrences)
HTOP outcome
PHP-fpm: pool www, sometimes consumes 100% of CPU and memory
NodePing Alert (Receiving continuous notification in day time)
failed the HTTP check. It is down as of Thu Apr 30 2020 12:29:09 GMT-0700.Timeout.
HTTP is back up after being down for 4 minutes as of Thu Apr 30 2020 23:28:19 GMT-0700.
Nginx.conf file
user www;
worker_processes 2;
#error_log logs/error.log;
#error_log logs/error.log notice;
error_log /var/log/nginx/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
client_header_timeout 3000;
client_body_timeout 3000;
fastcgi_read_timeout 3000;
client_max_body_size 32m;
fastcgi_buffers 8 128k;
fastcgi_buffer_size 128k;
server_name_in_redirect on;
server_names_hash_bucket_size 64;
server_names_hash_max_size 8192;
#access_log logs/access.log main;
access_log /var/log/nginx/access.log;
sendfile on;
keepalive_timeout 65;
gzip on;
gzip_vary on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
gzip_disable "MSIE [1-6]\.";
include /etc/nginx/ssl/*.conf;
server {
listen 80;
autoindex off;
server_name localhost;
add_header X-Frame-Options "SAMEORIGIN";
root /usr/local/www/html/webroot;
index index.html index.php;
# redirect server error pages to the static page /50x.html
location / {
# try_files $uri $uri/ /index.php?$uri&$args;
# set $new_uri $uri;
# try_files $uri $uri/ /index.php?$args;
try_files $uri $uri?$args $uri/ /index.php?$uri&$args /index.php?$args;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
# NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
include /etc/nginx/fastcgi_params;
#fastcgi_param PATH_INFO $new_uri;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
#fastcgi_pass /var/run/php5-fpm.sock;
fastcgi_read_timeout 300;
}
location = /favicon.ico {
log_not_found off;
access_log off;
}
location = /robots.txt {
log_not_found off;
access_log off;
}
error_page 500 502 503 504 /50x.html;
location = /50.html {
root /etc/nginx/html;
}
location ~ /(\.ht|\.user.ini|\.git|\.hg|\.bzr|\.svn) {
deny all;
}
}
# HTTPS server
#
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name *.XXXX.com;
ssl on;
ssl_certificate /XXXXX.crt;
ssl_certificate_key /XXXXX.key;
ssl_session_timeout 5m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1.1 TLSv1.2; # Dropping
root /usr/local/www/html/webroot;
index index.html index.php;
# redirect server error pages to the static page /50x.html
location / {
# try_files $uri $uri/ /index.php?$uri&$args;
# set $new_uri $uri;
# try_files $uri $uri/ /index.php?$args;
try_files $uri $uri?$args $uri/ /index.php?$uri&$args /index.php?$args;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
# NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
include /etc/nginx/fastcgi_params;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_read_timeout 3000;
}
location = /favicon.ico { log_not_found off; access_log off;}
location = /robots.txt { log_not_found off; access_log off;}
location ~ /.well-known { allow all; }
}
##
# Cache Proxy
##
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=STATIC:10m inactive=24h max_size=512m;
proxy_temp_path /var/tmp;
}
below are Some CPU utilization screens from AWS console
Above are some of my findings to found the issue, but I don't know what is causing poor server performance. Please suggest
UPDATE
I observe that in business hours (TTFB) is taking too much time (20 -25 sec)
I check the log of Mysql queries running on that page took Total Time: 1441 ms
So, Something else taking up time to load the page.
HTOP outcome at that time