I have a PHP application using these components:
Apache 2.2.3-31 on Centos 5.4
PHP 5.2.10
Xdebug 2.0.5 with Remote Debugging enabled
APC 3.0.19
Doctrine ORM for PHP 1.2.1 using Query Caching and Results Caching via APC
MySQL 5.0.77 using Query Caching
I've noticed that when I start up Apache, I eventually end up 10 child processes. As time goes on, each process will grow in memory until each one approaches 10% of available memory, which begins to slow the server to a crawl since together they grow to take up 100% of memory.
Here is a snapshot of my top output:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1471 apache 16 0 626m 201m 18m S 0.0 10.2 1:11.02 httpd
1470 apache 16 0 622m 198m 18m S 0.0 10.1 1:14.49 httpd
1469 apache 16 0 619m 197m 18m S 0.0 10.0 1:11.98 httpd
1462 apache 18 0 622m 197m 18m S 0.0 10.0 1:11.27 httpd
1460 apache 15 0 622m 195m 18m S 0.0 10.0 1:12.73 httpd
1459 apache 16 0 618m 191m 18m S 0.0 9.7 1:13.00 httpd
1461 apache 18 0 616m 190m 18m S 0.0 9.7 1:14.09 httpd
1468 apache 18 0 613m 190m 18m S 0.0 9.7 1:12.67 httpd
7919 apache 18 0 116m 75m 15m S 0.0 3.8 0:19.86 httpd
9486 apache 16 0 97.7m 56m 14m S 0.0 2.9 0:13.51 httpd
I have no long-running scripts (they all terminate eventually, the longest being maybe 2 minutes long), and I am working under the assumption that once each script terminates, the memory it uses gets deallocated. (Maybe someone can correct me on that).
My hunch is that it could be APC, since it stores data between requests, but at the same time, it seems weird that it would store data inside the httpd process.
How can I track down which part of my app is causing the memory leak?
What tools can I use to see how the memory usage is growing inside the httpd process and what is contributing to it?
My hunch is that it could be APC, since it stores data between requests, but at the same time, it seems weird that it would store data inside the httpd process.
What's weird about that? That's exactly what APC does. The memory is shared between all httpd processes, though, so it's not as bad as it sounds. See Where does APC store its opcode and user variable cache? for details.
Related
I will have an event with 3k users on an app (php base).
I launch several instances in the cloud and install LAMP on it.[to make load test and choose on for the event]
On Ubuntu 18
I enable mpm_event and php7.4-fpm, (which seems to be the better configuration for high traffic with apache and php app).
I use this post which explain how tune your conf.
Like this :
Here apache2 mpm event conf :
<IfModule mpm_*_module>
ServerLimit (Total RAM - Memory used for Linux, DB, etc.) / process size
StartServers (Number of Cores)
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers (Total RAM - Memory used for Linux, DB, etc.) / process size
MaxConnectionsPerChild 1000
</IfModule>
Here php7.4-fpm :
pm = dynamic
pm.max_children (total RAM - (DB etc) / process size)
pm.start_servers (cpu cores * 4)
pm.min_spare_servers (cpu cores * 2)
pm.max_spare_servers (cpu cores * 4)
pm.max_requests 1000
My goal is : even if I rely of these method, I would saw some metric like :
--> You have too many thread (from apache worker or from phpfpm) unused open
--> All your thread (from apache worker or from phpfpm) are already busy and use
I already test: htop, glance, vmstat, sar to check io, cpu, ram but even with that it's not clear to me :
Does my configuration is good for this machine with this load or should I increase/decrease something?
Then I could be sure these configuration are good and start other subject : CDN, cache ...
How do you manage this ?
thanks by advance,
As you noted, this depends on your script(s). We have this dynamically adjusted in our deploy scripts based on the server(s) being rolled up.
The following script is based on running Apache, on Centos, on AWS infrastructure but could easily be adapted to what you are using.
Basically:
set the size of apache processes
set the size of php process
scripts gets available memory, cores and does some crunching and then modifies the config.
we run this as part of stack roll up
Primary Source / Based on:
https://medium.com/#sbuckpesch/apache2-and-php-fpm-performance-optimization-step-by-step-guide-1bfecf161534
Steps:
Calculate process size
You need to know how many processes can run on your machine. So calculate the process size of your main CPU/memory drivers is necessary.
cd /tmp
curl https://raw.githubusercontent.com/pixelb/ps_mem/master/ps_mem.py --output ps_mem.py
chmod a+x ps_mem.py
sudo python ps_mem.py
# Sample numbers:
# 28.4 MiB + 103.0 KiB = 28.5 MiB memcached
# 34.7 MiB + 9.5 KiB = 34.7 MiB amazon-cloudwatch-agent
# 24.8 MiB + 18.0 MiB = 42.8 MiB httpd (15)
# 69.1 MiB + 7.0 MiB = 76.0 MiB php (2)
# 228.2 MiB + 46.0 MiB = 274.3 MiB php-fpm (36)
Here you can see that there are 15 httpd processes, consuming a total of 43MiB, so each Apache process is using roughly 3MiB of RAM.
The php-fpm process will use about 7.6MiB.
Calculate Apache MaxRequestWorkers
To be safe though, reserve 15% of memory for all other processes (in my case ~1.2GiB) and round up apache process size to 3MiB.
MaxRequestWorkers = (Total RAM - Memory used for Linux, DB, etc.) / process size
MaxRequestWorkers = (8000MB - 1200MB) / 3MB = 2,266
Calculate php-fpm max-children
To be safe though, reserve 1 GiB for all other processes and round up php process size to 8MiB.
max_children = (Total RAM - Memory used for Linux, DB, etc.) / process size
max_children = (8000MB - 1200MB) / 8MB = 850
Here is the script we use, on roll up.
#!/bin/bash
# Creates a configuration script to run once final servers are up.
PROCESS_SIZE_APACHE_MB=3
PROCESS_SIZE_PHP_MB=8
# Get some values from the server
MEMORY_KB=`grep MemTotal /proc/meminfo | awk '"'"'{print $2}'"'"'`
MEMORY_MB=$(($MEMORY_KB / 1024))
MEMORY_AVAILABLE_MB=$(($MEMORY_KB / 1178))
NUM_CORES=`nproc --all`
echo "Memory: $MEMORY_MB MB"
echo "Memory Available: $MEMORY_AVAILABLE_MB MB"
echo "Num Cores $NUM_CORES"
#Now do some calculations
SERVER_LIMIT=$(($MEMORY_AVAILABLE_MB / $PROCESS_SIZE_APACHE_MB))
echo "HTTP MPM Server Limit: $SERVER_LIMIT"
#Convert Apache from mpm-prefork to mpm-worker
#Set params
#<IfModule mpm_*_module>
# ServerLimit (Total RAM - Memory used for Linux, DB, etc.) / process size
# StartServers (Number of Cores)
# MinSpareThreads 25
# MaxSpareThreads 75
# ThreadLimit 64
# ThreadsPerChild 25
# MaxRequestWorkers (Total RAM - Memory used for Linux, DB, etc.) / process size
# MaxConnectionsPerChild 1000
# </IfModule>
# /etc/httpd/conf.modules.d/00-mpm.conf
echo "
# LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
# LoadModule mpm_worker_module modules/mod_mpm_worker.so
LoadModule mpm_event_module modules/mod_mpm_event.so
<IfModule mpm_*_module>
ServerLimit $SERVER_LIMIT
StartServers $NUM_CORES
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers $SERVER_LIMIT
MaxConnectionsPerChild 1000
</IfModule>
" > /etc/httpd/conf.modules.d/00-mpm.conf
# Configure the workers
# pm = dynamic
# pm.max_children (total RAM - (DB etc) / process size) = 850
# pm.start_servers (cpu cores * 4)
# pm.min_spare_servers (cpu cores * 2)
# pm.max_spare_servers (cpu cores * 4)
# pm.max_requests 1000
MAX_CHILDREN=$(($MEMORY_AVAILABLE_MB / $PROCESS_SIZE_PHP_MB))
echo "Max Children: $MAX_CHILDREN"
NUM_START_SERVERS=$(($NUM_CORES * 4))
NUM_MIN_SPARE_SERVERS=$(($NUM_CORES * 2))
NUM_MAX_SPARE_SERVERS=$(($NUM_CORES * 4))
sed -c -i "s/^;*pm.max_children.*/pm.max_children = $MAX_CHILDREN/" /etc/php- fpm.d/www.conf
sed -c -i "s/^;*pm.start_servers.*/pm.start_servers = $NUM_START_SERVERS/" /etc/php- fpm.d/www.conf
sed -c -i "s/^;*pm.min_spare_servers.*/pm.min_spare_servers = $NUM_MIN_SPARE_SERVERS/" /etc/php-fpm.d/www.conf
sed -c -i "s/^;*pm.max_spare_servers.*/pm.max_spare_servers = $NUM_MAX_SPARE_SERVERS/" /etc/php-fpm.d/www.conf
sed -c -i "s/^;*pm.max_requests = 500.*/pm.max_requests = 1000/" /etc/php-> fpm.d/www.conf
No tool will give you that kind of metric because the best configuration depends greatly on your php scripts. If you have 4 cores and each request consumes 100% of one core for 1 second, the server will handle 4 request per second in the best case regardless of your mpm and php configuration. The type of hardware you have is also important. Some CPUs perform multiple times better than others.
Since you are using php_fpm, the apache mpm configuration will have little effect on performance. You just need to make sure the server doesn't crash with too many requests and have more apache threads than php processes. Note that the RAM is not the only thing that can make a server unreachable. Trying to execute more process than the CPU can handle will increase the load and the number of context switches, decrease the CPU cache efficiency and result in even lower performance.
The ideal number of php processes depends on how your scripts use CPU and other resources. If each script uses 50% of the time with I/O operations for example, 2 processes per core may be ideal. Assuming that those I/O operations can be done in parallel without blocking each other.
You'll also need to take into account the amount of resources used by other processes such as the DB. SQL databases can easily use more resources than the php scripts themselves.
Spare Servers and Spare Threads are the number of processes/threads that can be idle waiting for work. Creating threads takes time, so it's better to have them ready when a request arrives. The downside is that those threads will consume resources such as RAM even when idle, so you'll want to keep just enough of them alive. Both apache and php_fpm will handle this automatically. The number of idle threads will be reduced and increased as needed, but remain between the minimum and maximum values set in the configuration. Note that not all apache threads will serve php files as some requests may be fetching static files, therefore you should always have more apache threads than php processes.
Start Server and Start Threads represents just the number of processes/threads created during the startup. This have almost no effect on performance since the number of threads will be immediately increased or reduced to fit the values of Spare Threads.
MaxConnectionsPerChild and max_requests are just the maximum amount of requests executed during the process/thread life. Unless you have memory leaks, you won't need to tune those values.
I'm having a problem when CPU is 100% usage: PHP-FPM does not respond, even the processes using 100% of CPU are running in lowest priority (+19).
Let me explain: my site triggers an audio converter (ffmpeg) using inotify + a bash script. The converter is running using "nice -n 19". So, CPU usage is 100% but, in theory, PHP-FPM has priority (default 0 priority).
What happens: only PHP freezes. Apache can respond to files like JPG/CSS/JS. Linux stable, other SWs too. All .php returns "504 Gateway Timeout" (browser, Apache response). Apache opens a lot of php-fpm instances and keep them open.
30585 fedora 19 0.0 0.3 0:00.00 snippeter
30586 fedora 19 25.0 2.8 0:00.10 ffmpeg
15640 fedora 0 0.0 1.0 0:00.12 php-fpm
17174 fedora 0 0.0 1.0 0:00.10 php-fpm
20583 fedora 0 0.0 0.9 0:00.00 php-fpm
21072 fedora 0 0.0 0.9 0:00.00 php-fpm
21309 fedora 0 0.0 0.9 0:00.00 php-fpm
22601 fedora 0 0.0 0.9 0:00.00 php-fpm
23172 fedora 0 0.0 0.9 0:00.00 php-fpm
23277 fedora 0 0.0 0.9 0:00.00 php-fpm
24170 fedora 0 0.0 0.9 0:00.00 php-fpm
24823 fedora 0 0.0 0.9 0:00.00 php-fpm
24916 fedora 0 0.0 0.9 0:00.00 php-fpm
26589 fedora 0 0.0 0.9 0:00.00 php-fpm
%Cpu(s): 8.5 us, 4.5 sy, 85.6 ni, 0.0 id, 0.5 wa, 1.0 hi, 0.0 si, 0.0 st
Changing php-fpm priority (renice), no difference.
I don't know what to do... I want to run processes in background, even using 100% of CPU, but keeps PHP+Apache running normally.
Server: Fedora 29, PHP 7.2.
Found a solution!
The problem is not related to 100% CPU usage, or just one CPU. Since I have a loop to open each audio converter (bash file), and wait it to finish to run the next one, the PHP session is locked! So, when another PHP script, in the same session, try to execute, it cannot read the session file, so, PHP freezes (waiting to have access to the session file, until a "Gateway Timeout" error). That's why only PHP freezes, not other softwares.
To fix, I had to run this before the main loop:
session_write_close();
session_start(['read_and_close'=>true]);
So, it writes and closes the session file, and read it again once, without locking it.
This only works in PHP 7+ and, obviously, you cannot write any session data. Or, after the loop, just open the session again with session_start().
I've an Ubuntu Linux 16.04.1 server, running a wordpress blog my server hardware is pretty ok for this task but I found that apache2 is loading too much memory.
After i JUST REBOOT the server the OS shows me this consume
2215 www-data 563.27 MB /usr/sbin/apache2 -k start
2216 www-data 563.27 MB /usr/sbin/apache2 -k start
2217 www-data 563.27 MB /usr/sbin/apache2 -k start
It's a pretty high value for a server before any request, I found the config file setting the number of spare servers to 3 after start up and it is okay.
but the amount of memory each server loads is what making me confuse and i didnt found any configuration to set this value or a min or max.
Does anyone know the default value for an apache php mysql running a wordpress?
are there any config file i can check for understand why such high memory?
I have deployed my PHP API on a Linux based server. It's users are 40000-50000. Server has Disk space 20GB and out of that, 15GB is used. I have used MySQL for Database. GZip is enabled and the response JSON on http://example.com/getData is nearly equal to 100KB. Initially, it used to take 2-3 seconds on each getData call (which includes querying from data table and processing on PHP), but now it is taking more than 15 seconds for a single request. I have checked the CPU usage by TOP command in linux. This is the CPU status:
top - 09:30:44 up 14:44, 3 users, load average: 1.16, 2.54, 2.28
Tasks: 222 total, 1 running, 221 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.2%us, 1.5%sy, 0.0%ni, 92.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.2%st
Mem: 2051652k total, 1576444k used, 475208k free, 204408k buffers
Swap: 0k total, 0k used, 0k free, 456548k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15942 www-data 20 0 320m 14m 6480 S 1 0.7 0:04.50 apache2
15356 mysql 20 0 1320m 52m 7448 S 1 2.6 1:00.19 mysqld
15973 www-data 20 0 321m 15m 6444 S 1 0.8 0:06.75 apache2
15811 www-data 20 0 318m 12m 5648 S 1 0.6 0:04.33 apache2
28989 root 20 0 96716 3928 2948 S 1 0.2 0:00.03 sshd
15806 www-data 20 0 323m 17m 6756 S 0 0.9 0:04.10 apache2
15807 www-data 20 0 321m 16m 6708 S 0 0.8 0:04.83 apache2
15817 www-data 20 0 322m 16m 6416 S 0 0.8 0:06.19 apache2
15832 www-data 20 0 320m 14m 6416 S 0 0.7 0:04.11 apache2
15838 www-data 20 0 320m 14m 6416 S 0 0.7 0:04.05 apache2
15849 www-data 20 0 321m 15m 6416 S 0 0.8 0:06.15 apache2
15851 www-data 20 0 321m 15m 6424 S 0 0.8 0:04.26 apache2
15853 www-data 20 0 320m 15m 6740 S 0 0.8 0:05.45 apache2
15856 www-data 20 0 323m 18m 6644 S 0 0.9 0:04.68 apache2
All the queries running time is less than 0.01 second and the microtime difference between end and start of the getData function is 0.51.
What may be the reasons of this slow response on browser?
I bet for DNS lookups in apache logs. Verify “HostnameLookups” are off in your apache config.
It could probably be the logs that is stored in your system , it holds records of application activities ex time, date and much more , that's a major thing in system logs as it reduces the performance of any application activities.
I have set up FastCGI to run my PHP site, and I found that when I generate a big file using the site, only one php-cgi process is working and its CPU usage is 100%. But the total CPU usage is only 25%.
Here is the "top -u user" output.
top - 13:50:35 up 3:58, 3 users, load average: 0.07, 0.20, 0.32
Tasks: 116 total, 1 running, 115 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.0%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 35126760k total, 1090160k used, 34036600k free, 60644k buffers
Swap: 0k total, 0k used, 0k free, 289816k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7570 user 20 0 232m 13m 10m S 100.0 0.3 0:00.03 php-cgi
7571 user 20 0 234m 16m 10m S 0.0 0.0 0:00.22 php-cgi
7572 user 20 0 304m 86m 10m S 0.0 0.3 1:37.49 php-cgi
7573 user 20 0 234m 15m 9228 S 0.0 0.0 0:00.16 php-cgi
7574 user 20 0 232m 13m 9528 S 0.0 0.0 0:00.40 php-cgi
7594 user 20 0 232m 13m 10m S 0.0 0.0 0:00.02 php-cgi
7595 user 20 0 234m 14m 9492 S 0.0 0.0 0:00.09 php-cgi
7596 user 20 0 232m 13m 9184 S 0.0 0.0 0:00.06 php-cgi
7597 user 20 0 234m 15m 9500 S 0.0 0.0 0:00.03 php-cgi
7598 user 20 0 232m 10m 7116 S 0.0 0.0 0:00.02 php-cgi
7599 user 20 0 232m 13m 10m S 0.0 0.0 0:00.02 php-cgi
7601 user 20 0 326m 33m 11m S 0.0 0.1 1:58.60 php-cgi
7602 user 20 0 234m 14m 8988 S 0.0 0.0 0:00.14 php-cgi
7603 user 20 0 234m 12m 6800 S 0.0 0.0 0:00.15 php-cgi
7604 user 20 0 308m 13m 9848 S 0.0 0.0 0:00.13 php-cgi
The server is AWS m2.x2large instance and it got 4 vCPU (4 cores maybe). And now the FastCGI seems to only use one core.
I have no idea about how to set up FastCGI to let it use all 4 vCPU.
My current settings is FastCGI + suEXEC + PHP 5.5.11 + apache2.
Here is teh FastCGI configurations in apache:
FcgidIPCDir /usr/local/apache2/logs/mod_fcgid
FcgidProcessTableFile /usr/local/apache2/logs/mod_fcgid/fcgid_shm
FcgidIOTimeout 7200
FcgidMaxRequestLen 2097152
FcgidMaxProcesses 100
FcgidMaxProcessesPerClass 100
FcgidMaxRequestsPerProcess 5000
FcgidMaxRequestInMem 20971520
FcgidZombieScanInterval 5
FcgidIdleScanInterval 120
FcgidProcessLifeTime 3600
Here is the /cgi-bin/php.fcgi settings:
#!/bin/bash
# Shell Script To Run PHP5 using mod_fastcgi under Apache 2.x
### Set PATH ###
PHP_CGI=/usr/bin/php-cgi
PHP_FCGI_CHILDREN=4
PHP_FCGI_MAX_REQUESTS=5000
### no editing below ###
export PHP_FCGI_CHILDREN
export PHP_FCGI_MAX_REQUESTS
exec $PHP_CGI
Please help me on this.
Thank you very much in advance!
The process is using 100% of one core, which is 25% of your total processing power (4 cores). There isn't a way for a single thread to use more than 1 core, but it looks like FastCGI can be configured to use multiple threads. Refer to this question for more details.
Note that 1 request can only result in 1 thread, so if this is handling a single request, there doesn't seem to be a way to have that request handled in multiple threads.