I will have an event with 3k users on an app (php base).
I launch several instances in the cloud and install LAMP on it.[to make load test and choose on for the event]
On Ubuntu 18
I enable mpm_event and php7.4-fpm, (which seems to be the better configuration for high traffic with apache and php app).
I use this post which explain how tune your conf.
Like this :
Here apache2 mpm event conf :
<IfModule mpm_*_module>
ServerLimit (Total RAM - Memory used for Linux, DB, etc.) / process size
StartServers (Number of Cores)
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers (Total RAM - Memory used for Linux, DB, etc.) / process size
MaxConnectionsPerChild 1000
</IfModule>
Here php7.4-fpm :
pm = dynamic
pm.max_children (total RAM - (DB etc) / process size)
pm.start_servers (cpu cores * 4)
pm.min_spare_servers (cpu cores * 2)
pm.max_spare_servers (cpu cores * 4)
pm.max_requests 1000
My goal is : even if I rely of these method, I would saw some metric like :
--> You have too many thread (from apache worker or from phpfpm) unused open
--> All your thread (from apache worker or from phpfpm) are already busy and use
I already test: htop, glance, vmstat, sar to check io, cpu, ram but even with that it's not clear to me :
Does my configuration is good for this machine with this load or should I increase/decrease something?
Then I could be sure these configuration are good and start other subject : CDN, cache ...
How do you manage this ?
thanks by advance,
As you noted, this depends on your script(s). We have this dynamically adjusted in our deploy scripts based on the server(s) being rolled up.
The following script is based on running Apache, on Centos, on AWS infrastructure but could easily be adapted to what you are using.
Basically:
set the size of apache processes
set the size of php process
scripts gets available memory, cores and does some crunching and then modifies the config.
we run this as part of stack roll up
Primary Source / Based on:
https://medium.com/#sbuckpesch/apache2-and-php-fpm-performance-optimization-step-by-step-guide-1bfecf161534
Steps:
Calculate process size
You need to know how many processes can run on your machine. So calculate the process size of your main CPU/memory drivers is necessary.
cd /tmp
curl https://raw.githubusercontent.com/pixelb/ps_mem/master/ps_mem.py --output ps_mem.py
chmod a+x ps_mem.py
sudo python ps_mem.py
# Sample numbers:
# 28.4 MiB + 103.0 KiB = 28.5 MiB memcached
# 34.7 MiB + 9.5 KiB = 34.7 MiB amazon-cloudwatch-agent
# 24.8 MiB + 18.0 MiB = 42.8 MiB httpd (15)
# 69.1 MiB + 7.0 MiB = 76.0 MiB php (2)
# 228.2 MiB + 46.0 MiB = 274.3 MiB php-fpm (36)
Here you can see that there are 15 httpd processes, consuming a total of 43MiB, so each Apache process is using roughly 3MiB of RAM.
The php-fpm process will use about 7.6MiB.
Calculate Apache MaxRequestWorkers
To be safe though, reserve 15% of memory for all other processes (in my case ~1.2GiB) and round up apache process size to 3MiB.
MaxRequestWorkers = (Total RAM - Memory used for Linux, DB, etc.) / process size
MaxRequestWorkers = (8000MB - 1200MB) / 3MB = 2,266
Calculate php-fpm max-children
To be safe though, reserve 1 GiB for all other processes and round up php process size to 8MiB.
max_children = (Total RAM - Memory used for Linux, DB, etc.) / process size
max_children = (8000MB - 1200MB) / 8MB = 850
Here is the script we use, on roll up.
#!/bin/bash
# Creates a configuration script to run once final servers are up.
PROCESS_SIZE_APACHE_MB=3
PROCESS_SIZE_PHP_MB=8
# Get some values from the server
MEMORY_KB=`grep MemTotal /proc/meminfo | awk '"'"'{print $2}'"'"'`
MEMORY_MB=$(($MEMORY_KB / 1024))
MEMORY_AVAILABLE_MB=$(($MEMORY_KB / 1178))
NUM_CORES=`nproc --all`
echo "Memory: $MEMORY_MB MB"
echo "Memory Available: $MEMORY_AVAILABLE_MB MB"
echo "Num Cores $NUM_CORES"
#Now do some calculations
SERVER_LIMIT=$(($MEMORY_AVAILABLE_MB / $PROCESS_SIZE_APACHE_MB))
echo "HTTP MPM Server Limit: $SERVER_LIMIT"
#Convert Apache from mpm-prefork to mpm-worker
#Set params
#<IfModule mpm_*_module>
# ServerLimit (Total RAM - Memory used for Linux, DB, etc.) / process size
# StartServers (Number of Cores)
# MinSpareThreads 25
# MaxSpareThreads 75
# ThreadLimit 64
# ThreadsPerChild 25
# MaxRequestWorkers (Total RAM - Memory used for Linux, DB, etc.) / process size
# MaxConnectionsPerChild 1000
# </IfModule>
# /etc/httpd/conf.modules.d/00-mpm.conf
echo "
# LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
# LoadModule mpm_worker_module modules/mod_mpm_worker.so
LoadModule mpm_event_module modules/mod_mpm_event.so
<IfModule mpm_*_module>
ServerLimit $SERVER_LIMIT
StartServers $NUM_CORES
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers $SERVER_LIMIT
MaxConnectionsPerChild 1000
</IfModule>
" > /etc/httpd/conf.modules.d/00-mpm.conf
# Configure the workers
# pm = dynamic
# pm.max_children (total RAM - (DB etc) / process size) = 850
# pm.start_servers (cpu cores * 4)
# pm.min_spare_servers (cpu cores * 2)
# pm.max_spare_servers (cpu cores * 4)
# pm.max_requests 1000
MAX_CHILDREN=$(($MEMORY_AVAILABLE_MB / $PROCESS_SIZE_PHP_MB))
echo "Max Children: $MAX_CHILDREN"
NUM_START_SERVERS=$(($NUM_CORES * 4))
NUM_MIN_SPARE_SERVERS=$(($NUM_CORES * 2))
NUM_MAX_SPARE_SERVERS=$(($NUM_CORES * 4))
sed -c -i "s/^;*pm.max_children.*/pm.max_children = $MAX_CHILDREN/" /etc/php- fpm.d/www.conf
sed -c -i "s/^;*pm.start_servers.*/pm.start_servers = $NUM_START_SERVERS/" /etc/php- fpm.d/www.conf
sed -c -i "s/^;*pm.min_spare_servers.*/pm.min_spare_servers = $NUM_MIN_SPARE_SERVERS/" /etc/php-fpm.d/www.conf
sed -c -i "s/^;*pm.max_spare_servers.*/pm.max_spare_servers = $NUM_MAX_SPARE_SERVERS/" /etc/php-fpm.d/www.conf
sed -c -i "s/^;*pm.max_requests = 500.*/pm.max_requests = 1000/" /etc/php-> fpm.d/www.conf
No tool will give you that kind of metric because the best configuration depends greatly on your php scripts. If you have 4 cores and each request consumes 100% of one core for 1 second, the server will handle 4 request per second in the best case regardless of your mpm and php configuration. The type of hardware you have is also important. Some CPUs perform multiple times better than others.
Since you are using php_fpm, the apache mpm configuration will have little effect on performance. You just need to make sure the server doesn't crash with too many requests and have more apache threads than php processes. Note that the RAM is not the only thing that can make a server unreachable. Trying to execute more process than the CPU can handle will increase the load and the number of context switches, decrease the CPU cache efficiency and result in even lower performance.
The ideal number of php processes depends on how your scripts use CPU and other resources. If each script uses 50% of the time with I/O operations for example, 2 processes per core may be ideal. Assuming that those I/O operations can be done in parallel without blocking each other.
You'll also need to take into account the amount of resources used by other processes such as the DB. SQL databases can easily use more resources than the php scripts themselves.
Spare Servers and Spare Threads are the number of processes/threads that can be idle waiting for work. Creating threads takes time, so it's better to have them ready when a request arrives. The downside is that those threads will consume resources such as RAM even when idle, so you'll want to keep just enough of them alive. Both apache and php_fpm will handle this automatically. The number of idle threads will be reduced and increased as needed, but remain between the minimum and maximum values set in the configuration. Note that not all apache threads will serve php files as some requests may be fetching static files, therefore you should always have more apache threads than php processes.
Start Server and Start Threads represents just the number of processes/threads created during the startup. This have almost no effect on performance since the number of threads will be immediately increased or reduced to fit the values of Spare Threads.
MaxConnectionsPerChild and max_requests are just the maximum amount of requests executed during the process/thread life. Unless you have memory leaks, you won't need to tune those values.
We are currently working on a small web app and we ran into an issue. The error keeps occuring and we got kind of stuck.
503 service Unavailable
No server is available to handle this request.
What we use:
PHP 7.3 fpm mode
apache2
VMWare, Machine with debian 9, 2cpu/2cores and 16gb RAM
We currently have about 70 users on the website. After about 24h the server keeps crashing.
after restarting the php service the website works again. So I'm guessing the issue has something to do with php and not with apache.
according to other sources this issue might be caused because of opcache or max sessions / max child config.
www.conf:
[www]
user = www-data
group = www-data
listen = /run/php/php7.3-fpm.sock
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 50
pm.start_servers = 12
pm.min_spare_servers = 8
pm.max_spare_servers = 24
pm.max_requests = 1000
php.ini: https://pastebin.com/JanEHinF
any ideas what could cause the error?
I have a relative busy web server with Nginx and PHP-FPM, and after a upgrade to PHP 7.2 I have noticed PHP-FPM processes are eating all my RAM after some time.
The server is a KVM machine (Centos 7) with 32Gb of RAM and 8Gb swap:
[root#www ~]# free
total used free shared buff/cache available
Mem: 32779736 18397204 239372 1508476 14143160 12417824
Swap: 8257532 167680 8089852
Here you can see the memory consumption:
https://justpaste.it/2vaqy
(I tried to paste it here, but Stackoverflow says my post looks spam)
As you can see in RSS colum, in 4th March each php-fpm process is using about ~1Gb of RAM, that makes a total memory comsuption (22 processes) of 22Gb.
Each PHP-FPM process ram usage is incrementing about ~300Mb per day.
This is my configuration:
[root#www ~]# cat /etc/php-fpm.conf | sed '/^;/d' | sed '/^$/d'
include=/etc/php-fpm.d/*.conf
[global]
pid = /run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/error.log
daemonize = yes
[root#www ~]# cat /etc/php-fpm.d/www.conf | sed '/^;/d' | sed '/^$/d'
[www]
user = www
group = www
listen = xxxxx/php-fpm.sock
listen.mode = 0666
listen.allowed_clients = 127.0.0.1
pm = dynamic
pm.max_children = 200
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
pm.status_path = /xxxx-fpm
slowlog = /var/log/php-fpm/www-slow.log
request_terminate_timeout = 5h
php_flag[display_errors] = off
php_admin_value[error_log] = xxxx/php-fpm-error.log
php_admin_flag[log_errors] = on
This is my phpinfo:
https://justpaste.it/50zin
Any idea on whats going on?
Thank you in advance. Regards.
This looks simple, if you have memory leaks, you must re-spawn php worker as many as possible, this can be configured with pm.max_requests option.
Start with 500, the more is the best performance you have but the more leak as well.
Having set the pool to static with max_children to 5 I would expect the metric "active processes" to be 5 or below. Sending 10 concurrent requests will have "active processes" report more than 5 (e.g. 10, 12, 25, ...).
Is this valid behaviour?
Pool configuration:
# grep -v ";" /usr/local/etc/php-fpm.d/www.conf | grep -Ev "^$"
[www]
user = www-data
group = www-data
listen = 127.0.0.1:9000
pm = static
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
pm.process_idle_timeout = 10s
pm.max_requests = 500
pm.status_path = /status
ping.response = pong
slowlog = log/$pool.log.slow
request_slowlog_timeout = 0
request_terminate_timeout = 0
Expected result:
Metric "active processes" from /status should be below 5.
Actual result:
Metric "active processes" from /status is above 5.
According to the PHP FPM documentation when pm is set to static the only setting that determines the number of child processes is pm.max_children. As you expect, that should determine the start and max number.
A couple things to consider:
After changing the configuration did you restart both PHP-FPM and NGINX ? Something like:
sudo service php-fpm5 restart
sudo service nginx restart
Is it possible you are altering the wrong config file (perhaps one that is in the wrong path?). You might try changing some other value and confirm that it is affecting the web server as intended.
Is it possible that you have multiple pools configured and that the /status page is giving you a report for multiple pools.
My site is working fine with Nginx + PHP-FPM but randomly it gives 502 error. Environment Details
OS - CetnOS 6
Nginx
PHP-FPM (php 5.4)
APC (Code Cache APC 3.1.13 beta)
Memcache (data cache)
In php-fpm
pm.max_children = 200
pm.start_servers = 40
pm.min_spare_servers = 30
pm.max_spare_servers = 50
pm.max_requests = 500
Also i am using TCP connection not socket.
If any body has any input please update me.
Thanks
Firstly reduce pm.max_children = 200 to pm.max_children = 50.
You will have to firstly increase the file limit of the system thereby allowing nginx and php-fpm to open a larger number of files. File limit has to be increased as in linux everything in the end is a file. So the more connections you open the more number of files will be required. In ubuntu the file limit configuration are done in /etc/security/limits.conf. You will need to locate this for CentOS.
Then try to increase the internal port range which can be used by php-fpm. Along with this tcp ports are generally associated with a timeout value before they are reused, reduce this timeout so that more ports are freed when their job is done.
Find detailed info here.
Addition:
In case if the error still persists try increasing the number of php-fpm worker processes to 100. Although it is not recommended to set the value so high as they consume addition memory.
pm.max_children = 100
pm.start_servers = 90
pm.min_spare_servers = 70
pm.max_spare_servers = 100
You can try out various values to get the optimum suited for your purpose.
The basic reason for a 502 is when nginx cannot forward or fails to forward the request to php-fpm. Increasing the number of php-fpm worker processes can be one of the way, thereby giving nginx more processes to forward the requests to.