PHP server change between Apache and Nginx: 3x performance loss - php

Our current infra works fine with Apache + PHP and MySQL managed elsewhere. I have not done any particular optimization on this server. These are the initial configurations. Except for memory_limit and execution_time.
I am in the process of changing our infra to something more modern with bigger server and Nginx + PHP-FPM.
While working on this problem, I realize that we have big performance losses using this new infra. It's a shame to have more powerful servers and it's slower.
I watched the server load and nothing. Memory and CPU are barely 2%
I pulled out my Inspector badge and began to eliminate guesswork. Obviously, my tests use exactly the same case, same framework, same version of our app, ...
Here are my tests:
I installed on newer Apache + PHP servers and in fact Apache is even slower than with Nginx (no surprise :)).
I created a simple "hello wok" PHP file to test if it could come from Nginx. So I am not using our framework which could be involved.
Everything is normal TTFB times are almost equivalent (~ 24ms). Nginx on the new infra is 2ms faster.
While going with our framework, I added a piece of basic code which amounts to doing a "Hello wok" but using our framework.
Ah there, we have a problem. Finally ! Old infra 61ms, New infra: 184ms.
The new infra is therefore 3x slower to load the framework.
Obviously, I did a lot of optimizing the existing code but seeing that, I'll never be able to optimize it and make it 3x faster.
In all cases: Why the same code takes 3x longer to run on different environments.
The only thing that occurs to me right now is a PHP and PHP-FPM configuration problem.
In these configurations, do you know of any reasons that could have such a slowdown?
Obviously I looked for and tested things to optimize but it's always the same.
Here is my php.ini file:
date.timezone = Europe/Paris
opcache.enable = 0
apc.enabled = 1
apc.enable_cli = 1
upload_max_filesize = 128M
post_max_size = 128M
realpath_cache_size=4096k
realpath_cache_ttl=7200
display_errors = Off
display_startup_errors = Off
access.log = /proc/self/fd/2
error_log = /proc/self/fd/2
error_reporting = E_ALL ^ E_STRICT
; output_buffering=4096
pm = dynamic
pm.max_children = 5
pm.start_servers = 3
pm.min_spare_servers = 2
pm.max_spare_servers = 4
pm.max_requests = 1000
max_execution_time=600
memory_limit = 2048M
opcache.enable = 1
opcache.enable_cli = 1
opcache.memory_consumption = 2048
; opcache.interned_strings_buffer = 32
; opcache.max_accelerated_files = 100000
; opcache.max_wasted_percentage = 15
opcache.revalidate_freq = 0
; opcache.validate_timestamps = 1
Do you have any advice ? recommendations ? ideas ?
thanks in advance
Are you doing well
EDIT #1
Server Information:
Old server:
VM with 1 processor / 2GB RAM
No database, just Apache + PHP
Framework specific cache system: file
$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Common KVM processor
cpu MHz : 3504.002
cache size : 4096 KB
[...]
New server:
Docker container on a 3 processor / 4GB RAM server
I do not have the speeds of each processor. This is vCPU.
No database
Container PHP-FPM + Container Nginx
Framework specific cache system: APC
processor : 0
vendor_id : AuthenticAMD
cpu family : 23
model : 1
model name : AMD EPYC 7281 16-Core Processor
stepping : 2
cpu MHz : 2096.062
cache size : 512 KB
[...]
processor : 1
vendor_id : AuthenticAMD
cpu family : 23
model : 1
model name : AMD EPYC 7281 16-Core Processor
stepping : 2
cpu MHz : 2096.062
cache size : 512 KB
[...]
processor : 2
vendor_id : AuthenticAMD
cpu family : 23
model : 1
model name : AMD EPYC 7281 16-Core Processor
stepping : 2
cpu MHz : 2096.062
cache size : 512 KB
[...]
Others :
The database is managed in a provider
The sessions are stored in the database

Despite what you will read all over the internet, under low loads, Apache+mod_php is significantly faster than nginx+php-fpm. The latter architecture offers better capacity and scaling. That does not explain the difference in performance reported here.
The only thing you have told us about the two infrastructures is that one is old and one is new. If all you know about them is that you clicked a button in a cloud control panel then it can be difficult to find out more (if this is a Linux VM, check /proc/cpuinfo and mpstat paying particular attention to the steal time). The metrics you measured by testing will be highly dependent on single thread performance which is not a good indicator of throughput on a multi-cpu machine or any machine under heavy load.
But the first place I would be looking is at the storage latency / increasing opcache.revalidate_freq.

Related

Compute Engine MYSQL Server CPU Strange

I couldn't think what else to title this strange problem.
We have a "Worker" Compute Engine which is a MySQL SLAVE. Its primary role is to process a large set of data and then place it back on the Master. All handled via a PHP Script.
Now the processing of data takes roughly 4 hours to complete. During this time we noticed the following CPU pattern.
What you can see above is the 50% solid CPU starts after a server reboot. Then after about 2 hours its starts to produce a ECG style pattern on the CPu. Around every 5/6 minutes CPU spikes to ~48% then drops over the 5 minutes.
My question is, why. Can anyoen please explain why. We ideally want this server to be Maxing out ots cpu at 100% (50% as there are 2 cores)
The spec of the server: 2 VCPU's with 7.5GB Memory.
As mentioned, if we can have this running full throttle it would be great. Below is the my.cnf
symbolic-links=0
max_connections=256
innodb_thread_concurrency = 0
innodb_additional_mem_pool_size = 1G
innodb_buffer_pool_size = 6G
innodb_flush_log_at_trx_commit = 1
innodb_io_capacity = 800
innodb_flush_method = O_DIRECT
innodb_log_file_size = 24M
query_cache_size = 1G
query_cache_limit = 512M
thread_cache_size = 32
key_buffer_size = 128M
max_allowed_packet = 64M
table_open_cache = 8000
table_definition_cache = 8000
sort_buffer_size = 128M
read_buffer_size = 8M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 128M
tmp_table_size = 256M
query_cache_type = 1
join_buffer_size = 256M
wait_timeout = 300
server-id = 2
relay-log = /var/log/mysql/mysql-relay-bin.log
log_bin = /var/log/mysql/mysql-bin.log
log-error=/var/log/mysqld.log
read-only = 1
innodb_flush_log_at_trx_commit=2
I have cleaned up the above to remove any configs with private information which are not relevant to performance.
UPDATE
I have noticed when the VPU starts dropping during the heartbeat section of the graph the PHP script is no longer running. This is impossible, as the script I know takes 4 hours. No errors, and after another 4 hours the data is where I expected it.
Changing innodb_io_capacity = 800 to 1500 will likely reduce your 4 hour elapsed time to process by raising the limit to what you know you can achieve with your slave processing.
For your 7.5G indicated environment, configuration has
innodb_additional_mem_pool_size=1G
innodb_buffer_pool_size=6G
query_cache_size=1G
so before you start, you are overcommitted.
Another angle to consider, with
max_connections=256
max_allowed_packet=64M
could on a fully busy 256 connections need 16GB + just for this function to survive.
It is unlikely max_allowed_packet at 64M is reasonable.
Changing read_rnd_buffer_size = 4M to SET GLOBAL read_rnd_buffer_size=16384; could be significant on your slave then 24 hours later on master. They can be different but if it is significant in reducing your 4 hours on the slave, implement on both instances. Let us know what this single change does for you, please.
The 50% cpu utilization you are seeing is the script maxing out the --- single core that it is capable of utilizing --- . As indicated by PressingOnAlways recently. You can not tune around limit in your running script.
For a more thorough analysis, provide from SLAVE AND MASTER
RAM size (nnG)
SHOW GLOBAL STATUS
SHOW GLOBAL VARIABLES
SHOW INNODB STATUS
CPU % is measured by all the cores - so 100% cpu usage == both cores maxing out. PHP by default runs in a single thread and does not utilize multi-cores. The 50% cpu utilization you are seeing is the script maxing out the single core that it is capable of utilizing.
In order to utilize 100% cpu, consider spawning 2 PHP scripts that work on 2 separate datasets - e.g. script 1 processes records 1-1000000, while script 2 processes 1000001-2000000.
Other option is to rewrite the script to utilize threads. You may want to consider changing the language altogether for something that is more conducive to threads, like Golang? Though this might not be necessary if the main work is done within mysql.
The other issue you're seeing when the graph is below 50% may be due to IO wait. It's hard to tell from a graph though, you may be having a data flow transfer bottleneck where your CPU isn't working and waiting while large bits of data is transferred.
Optimizing CPU utilization is an exercise in finding the bottlenecks and removing them - good luck.
'Monitoring Service' could enabled to periodically capture a 'health check' of your system since it appears to be on a 6 minute cycle when you see spikes.
SHOW GLOBAL STATUS LIKE 'Com_show_%status' may confirm activity of this nature.
Divide your com_show_%status counters by (uptime/3600) to get rate-per-hour.
10 times an hour would be every 6 minutes.

How to calculate used memory for php-fpm?

Suppose I have a php application that runs with php-fpm. Below are the options for php/php-fpm and opcache. How can I calculate how much memory is used by all of the components?
memory_limit=128M
...
opcache.memory_consumption=128M
...
[www]
pm = dynamic
pm.max_children = 2
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 2
Here is my thinking. php-fpm is starting 1 process with 128 MB and opcache 128MB = 256 MB. If php-fpm creates a new server/process we will have 2 processes with 256 MB and opcache 128 MB = 384 MB. Is that correct?
The background is to calculator the correct memory limits for a Kubernetes deployment.
If you're talking about how much memory the app uses, you can use something like
ps aux | grep -P "fpm|COMMAND"
Once you have that, 10MB for e.g., you know how many MB your FPM thread is using. This is your child.
Next you need to know how much memory your server is using. Open htop, sort by mem (shift + m), take all your top mem usages above 1MB, which are not php related, add them up, thats your current server usage (non php).
If you can load your server when you check this that helps, particularly if your running mysql, that uses alot of memory, so helps if you load your server a bit there when you do your check.
If you find for e.g. your 8GB server uses only 512MB mem, you can reserve some extra space for it, say 750MB, or 1GB. that leaves the remaining 7GB to divide by your app mem usage. 10MB/7GB = 700 threads/childs.

Big database optimisation

I created a web service that generates huge requests every seconds.
Sometimes the MYSQL service seem down few secondes and work again well.
The main table contain more than 4 420 115 entries since one month.
Storage engine : InnoDB
The server configuation :
CPU : Intel(R) Xeon(R) CPU D-1540 # 2.00GHz
Coeurs : 16
Cache : 12288KB
RAM : 4x 32Go
Disques : 2 x 480 Go
The my.cnf :
skip-external-locking
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam-recover = BACKUP
query_cache_limit = 1M
query_cache_size = 16M
Can I have advices to avoid this problem and increase the mysql performances.
Thank's
Sounds like it may be time to iterate on the database structure and application logic to reduce the amount of DB interactions. Another option is adding a memcached or redis layer between the application and the SQL database to provide near millisecond response time for read actions.

heavy Drupal 7 site performance issue

I am running a drupal 7 website on a Linux server (with 4 cores and 12GB RAM) with LEMP (nginx+php5-fpm+mysql).
The drupal installation has a large number of modules enabled(all of which are needed).
I also use apc + boost + memcache + authcache for caching. The caching seems to be working (i see pages being served from cache)
and the site has a reasonable response time.
I have run stress tests with the website running in a url like www-1.example.com. (www-1.example.com points to the ip of my webserver, let's say x.x.x.x)
and the results are fine (for up to 100 concurrent users)
The problem starts when I change the dns so that www.example.com also points to x.x.x.x. Then the cpu of my webserver (all 4 cores) reach 100% at short time.
I have been expereimenting with the following parameters on the www.conf file with no luck:
Configuration 1:
pm.max_children = 100
pm.start_servers = 20
pm.min_spare_servers = 10
pm.max_spare_servers = 20
pm.max_requests = 200
result: 100% cpu usage, low memory usage
Configuration final:
pm.max_children = 300
pm.start_servers = 20
pm.min_spare_servers = 10
pm.max_spare_servers = 20
pm.max_requests = 200
result: low cpu usage, 100% memory usage
Can anyone guide me to find the optimal comfiguration or has any idea on what can cause the 100% cpu usage?
How do i culculate the maximum number of concurrent usages that can run without problem based on server parameters?

Memory usage high on server compared to wamp

Lately my site (with 260000 posts, 12000 images, 2,360,987 mysql rows and 450.7 MiB size) is running slow and at times not loading for many mins
I installed this Debug bar plugin https://wordpress.org/plugins/debug-bar/
Memory usage
on server is: 174,319,288 bytes
Intel(R) Xeon(R) CPU E3-1230 V2 # 3.30GHz , 16 GB
(PHP: 5.5.23, MySQL: 5.6.23, Apache 2.4)
Even tried disabling all plugins it doesnt help much... it comes down 160-163,xxx,xxx bytes
on wamp is : 37,834,920 bytes
(PHP: 5.5.12, MySQL: 5.6.17)
Why the difference is huge? How to detect the problem?
Been using the following plugins
Acunetix WP Security
Akismet
Antispam Bee
CloudFlare
Contact Form 7
Custom Post Type UI
Debug Bar
Login LockDown
Redirection
Theme Test Drive
W3 Total Cache
WordPress SEO
WP-Optimize
WP Missed Schedule
my.cnf values for the above server are
[mysqld]
slow-query-log=1
long-query-time=1
slow-query-log-file="/var/log/mysql-slow.log"
default-storage-engine = MyISAM
local-infile = 0
innodb_buffer_pool_size = 1G
innodb_log_file_size = 256M
innodb_file_per_table=1
innodb_stats_on_metadata=0
max_connections=360
wait_timeout=60
connect_timeout = 15
thread_cache_size=20
thread_concurrency=8
key_buffer_size = 1024M
join_buffer_size = 2M
sort_buffer_size=1M
query_cache_limit=64M
query_cache_size=128M
query_cache_type=1
max_heap_table_size=32M
tmp_table_size=32MB
table_open_cache=1000
table_definition_cache=1024
open_files_limit=10000
max_allowed_packet=268435456
low_priority_updates=1
concurrent_insert=2
#port = 8881
#innodb_force_recovery=0
#innodb_purge_threads=0
The "server" has Apache; that accounts for some (all?) of the difference.
Windows and Unix handle memory differently, and measure it differently. So, the difference may be irrelevant.
The numbers you have quoted are not big; what is the problem?
"Tried restarting the server and checked it in initial moments" -- That's mostly irrelevant. Programs tend to grow over time, up to some limit. Let's see the values in "steady state" with a typical load.
You have enough RAM to cache the entire dataset in RAM. But, due to inactivity, probably most of the data is not touched, hence has not been read into cache.
"High" memory usage is when you are swapping. Actually, that is probably "too high". So, say, 90% is "high". Your numbers are nowhere near that.
innodb_buffer_pool_size=200M -- is not enough to hold the entire 450.7MB dataset, but, as I say, most of the data is probably not actively used.
Edit (after posting of settings)
table_cache=10M
That is terrible! You won't be opening 10 million tables. Set it to 1000.
max_heap_table_size=512M
tmp_table_size=512MB
Those are dangerous. If you have multiple connections, each needing a tmp table (because of a complex query), you could run out of memory fast. Set them down to 32M.
innodb_force_recovery=3
Comment out that line -- It is to be used once, then removed.
The rest of the settings look harmless for this discussion.

Categories