I have a shop on PHP platform (bad developed) where there are a lot of bad queries (long queries without indexes, order by rand(), dynamic counting, ..)
I do not have the possibility to change the queries for now, but I have to tune the server to stay alive.
I tried everything I know, I had very big databases, which I optimized with changing MySQL engine and tune it, but it is first project, where everything goes down.
Current situation:
1. Frontend (PHP) - AWS m4.large instance, Amazon Linux (latest), PHP 7.0 with opcache enabled (apache module), mod_pagespeed enabled with external Memcached on AWS Elasticache (t2.micro, 5% average load)
2. SQL - Percona DB with TokuDB engine on AWS c4.xlarge instance (50% average load)
I need to lower the instances, mainly the c4.xlarge, but if I switch down to c4.large, after a few hours there is a big overload.
Database has only 300MB of data, I tried query cache, InnoDB, XtraDB, ... with no success, and the traffic is also very low. PHP uses MySQLi extension.
Do you know any possible solution, how to lower the load without refactoring the queries?
Use of www.mysqlcalculator.com would be a quick way to get a brain check on about a dozen memory consumption factors in less than 2 minutes.
Would love to see your SHOW GLOBAL STATUS and SHOW GLOBAL VARIABLES, if you could get them posted to allow analysis of activity and configuration requests. 3 minutes of your GENERAL_LOG during a busy time would be interesting and could identify troublesome/time consuming SELECT's, etc.
Related
I'm running an app that interacts with a mysql database using the native Mysql PDO, I'm not using laravel or any framework.
So most of the APIs logic is to fetch from DB and/or to insert.
when running on production I can see high memory usage from mysql tasks, check below:
I have a couple of questions.
Is such high memory usage normal? and what's the best practice to manage a proper PHP-MysQL connection in a multi-threaded production-level app?
When an intensive query is running (fetching historical data to plot a graph), the CPU usage jumps to 100% until the query execution finishes it returns back to 2-3%. But during that time the system is completely paused.
I'm thinking of hardware based solutions, such as separating the db server from the app server (currently they both run on the same node) And managing a cluster and using read-only nodes.
But i'd like to know if there are other options, and what's the most efficient way to handle PHP-MySQL connections.
You could also check the query written if there are fully optimized, and connection are closed when not used.
Also you can reduce the load on the mysql server by balancing some work to php.
Also take a look at your query_cache_size, innodb_io_capacity, innodb_buffer_pool_size and max_connection setting in your my.cnf.
Also sometimes upgrading php and doing some caching on your apache can help reduce of ram uses.
https://phoenixnap.com/kb/improve-mysql-performance-tuning-optimization
Normally, innodb_buffer_pool_size is set to about 70% of available RAM, and MySQL will consume that, plus other space that it needs. Scratch that; you have only 1GB of RAM? The buffer_pool needs to be a smaller percentage. It seems that the current value is pretty good.
You top show only 37.2% of RAM in used by MySQL -- the many threads are sharing the same memory.
Sort by memory (in top, use < or >). Something else is consuming a bunch of RAM.
query_cache_size should be zero.
max_connections should be, say, 10 or 20.
CPU -- You have only one core? So a big query hogging the CPU is to be expected. It also says that I/O was not an issue. You could show us the query, plus SHOW CREATE TABLE and EXPLAIN SELECT...; perhaps we can suggest how to speed it up.
Lately my site has been getting about 2.5 million hits per day (on average). I record hits to each and every page (it's an adult site), so I'm able to have a Top 10 sort of thing that shows top Websites, Models, Galleries and Images. I record the hit, as well as the users IP so those individual sections only get incremented one time per user, every 24 hours. The problem with this is that it's updating the mysql database each hit. So of course, my site has started getting 504 errors.
I looked around and saw that memcached might be a solution. Store hits in memory and push to the database every X mins. I also saw some people suggest using MongoDB, which to my understanding is also a memory type storage. Would this be the way to go? Would you recommend memcached or MongoDB for what I'm trying to do? Or is this not the way to proceed because it just means more mysql calls in a shorter time frame (1 huge batch, say, every minute would mean 60 seconds worth of hits versus smaller batches every second).
I have both memcached and MongoDB installed on my server, so either is an option.
there may be much easier solutions to obtain better database performance without new software packages. the volumes you mention are not particularly large.
i'll list a just a few of many possibilities.
1. if you are on a version of mysql older than 5.6, then updating to 5.6+ will almost certainly yield a very significant improvement because the storage engine is much better for 5.6 and above.
2. if the busiest tables use a storage engine other than innodb, then switch to innodb. [you can do this with phpmyadmin]
3. get some help tuning buffer sizes in my.ini [it takes some skill] and/or increasing ram on the database server(s).
4. consider spreading the workload across more drives and/or switch part or all of the database to solid state drives [or better conventional drives]
5. if the database server(s) is/are memory or compute bound then bigger or more servers may be needed.
6. make sure the bottleneck is not external to the database server(s).
The way this site (stackoverflow.com) implements it is by maintaining in-memory data structure of question views which gets flushed to DB every 15 minutes or so. There is no need to stress DB by saving each hit - too much IO. This in-memory structure could be just within your application as a map of ip and hits/time or it could be in memcached. I don't think you really need memcached for this purpose.
So the general idea to do batch updates that you had is a good one.
Let me show what problem I'm dealing with:
website powered by Apache 2.2 + PHP 5.x + MySQL 5.1.x
peak traffic = 2.000 unique visitors/min = 5-8k pageviews/min
normal traffic = 2.000 unique visitors/day
website works well while under normal traffic
website lags while under peak traffic
my server cpu load is pretty big while under peak traffic (because of mysql/php processes), so my website is lagging.
Normal state: server response in 0.1-0.4 sec/pageview. PHP code is optimized to get and process all data from database and output HTML code within this time (call it server-response).
Peak traffic state: server response in 2-5 sec/pageview. And that's a bit longer response than I'm happy with. I don't want my visitors to wait so long for requested page.
What I'm doing now: My way to deal with this problem now is local cache system. I'm making local cache file (stored on disk) for about 10 minutes with cached SQL results - so I don't have to call the same sql query with every request.
My website is http://www.lechaton.cz/.
Is there any better way how to deal with peak traffic or optimize CPU utilization?
Thanks all for your time and advise!
I've been working through whole weekend and testing and testing and comparing methods and solutions.
Nginx solution (replace LAMP with LNMP)
I'm still using Apache2.2, but even using nginx (thanks to Bondye) there is not a big difference.
I've tried LNMP on debian wheezy, but with not a big difference from LAMP.
For static files nginx is faster in fact.
Nginx with PHP-FPM is a twice faster, that could be solution for some cases, but not solving my issues for 100%.
Tune-up your MySQL settings and MySQL queries
Tune-up your MySQL server with better caching and buffering. Also check your max-connections and memory usage.
But the most important is to optimize your queries for best performance. Even if your queries are best performing with mysql-cache, bigger traffic brings your server down within several minutes of big traffic. YOU HAVE TO CACHE your output!!
Tune-up apache
tune up your apache mpm-prefork
Reduce KeepAliveTimeout to max 5 seconds (default=30)
keep maxClients set to correct number (depends on your RAM, max-processes and max-servers in settings directive)
Final conclusion
1) content cache = rule #1
I've found the best solution is: cache as much as much content as you can. There is no reason why every user should generate same output if there is option to display cached content. It's much faster and it saves your resources.
2) nginx for static content
You can use nginx to perform best with static files and content, there is much lower cpu load with multiple processes.
With PHP-FPM your code speeds up twice (maybe a bit more). But I can't consider it as final solution.
3) test your website with benchmark tools I've used siege and apache benchmark (ab) and mysqlslap.
These steps helped me to reduce CPU load with my brand new server, speed up my server-response and balance my peak-traffic during big events.
Hope someone will find it helpful.
I am writing a PHP application which uses MySQL in the backend. I am expecting about 800 users a second to be hitting our servers, with requests coming from an iOS app.
The application is spread out over about 8 diffrenet PHP scripts which are doing very simple SELECT queries (occasionally with 1 join) and simple INSERT queries where I'm only inserting one row at a time (with less than 10kb of data per row on average). Theres about a 50/50 split between SELECTS and INSERTS.
The plan is to use Amazon Web Services and host the application on EC2s to spread the CPU load and RDS (with MySQL) to handle the database, but I'm aware RDS doesn't scale out, only up. So, before committing to an AWS solution, I need to benchmark my application on our development server (not a million miles of the medium RDS solution spec) to see roughly how many requests a second my application and MySQL can handle (for ballpark figures) - before doing an actual benchmark on AWS itself.
I believe I only really need to performance test the queries within the PHP, as EC2 should handle the CPU load, but I do need to see if / how RDS (MySQL) copes under that many users.
Any advice on how to handle this situation would be appreciated.
Thank-you in advance!
Have you considered using Apache Benchmark? Should do the job here. Also I've heard good things about siege but haven't tested yet.
If you have 800 user hits per second, it could be a good idea to consider sharding to begin with. Designing and implementing sharding right at the beginning will allow you to start with a small number of hosts, and then scale out more easily later. If you design for only one server, even if for now it will handle the load, pretty soon you will need to scale up and then it will be much more complex to switch to a sharding architecture when the application is already in production.
I have a LAMP application running fine, however the number of users are increasing each day. I don't want to be hit with a surprise one morning and find that everything broke because of overload. Is there a way to get a rough estimate of what capacity of the LAMP it is at?
I know that a full detailed report is many books worth of study but can I get some quick litmus test to see if things are running fine.
So say for the mysql component, how can I tell how much more load can it take? Is it at 30% capacity, 50%? etc.
Same for my apache. Although I have a feeling the DB will die before apache.
Perhaps my original was not too good, as English is not my native language. What I am really asking is a way to measure the current load. And then have a way to estimate based on the that load, how much further can I go before it fails. (And this should be done seperately for each component, mysqld, httpd)
ab is a bit annoying if your site needs cookies, etc, ab is too simple.
Basically, from my experience in fixing several imploding PHP websites, it usually goes like this :
1) People use MySQL
You can totally use MySQL, facebook and flickr do it (mysql fanboys love those) IF YOU KNOW THE GOTCHAS which are :
If you have a non-read-only MyISAM table and any query longer than 100 us (even selects) you are dead
On one site I fixed, the guy had rented a double-quad-core server because "his site needs the power". I look at his site, I look at my previous site with > 100K members and a torrent tracker which ran on a Via C7 micro-half-pizzabox server, and I tell him, your site runs fine on the Celeron 300 that's in my basement, and that's even overkill, I can rent it to you for half the price of your Xeon, lol.
It turned out that the guy was a good developer and a real nice guy but he sucked at MySQL, so his site had the typical Search Query From Hell that can kill any website :
10 search queries from hell per second (he had like 300K members on his illegal warez site)
search query from hell takes about 0.1 - 0.2 seconds
a little stream of concurrent updates to the same MyISAM table to spice things up
=> total serialization (MyISAM write locks) of all queries. 1 core 100%, 7 cores idle, loadavg > 1000 (yes he was using apache), page times > 30 seconds, the works.
Fix was easy : optimize the search query from hell, fix point 2) below, switch to InnoDB, switch to lighttpd. loadavg dropped to 0.02
2) UPDATEs
Noone is interested in page counters.
Issue 1 UPDATE for every page view and you are dead.
Add some MyISAM for more effects. Also a killer on InnoDB, not about locking, rather about sync disk IO waits.
3) FULLTEXT
MyISAM not usable for read-write tables because of locking.
MyISAM is as reliable as a ramdisk (in fact, less : you need an OS crash to corrupt a ramdisk, corrupting MyISAM tables just needs a MySQL crash or just hitting it too much concurrently, you'll get "unknown table engine error", I saw this many times)
FULLTEXT not available on InnoDB
Any insertion in a FULLTEXT index triggers almost a full index rebuild (when I inserted a
forum post it was rebuilding 400 MB of index)
==> If you need full text indexing, performance, and reliability, use Sphinx or Xapian.
I've not tried Sphinx (people say good things about it), but Xapian happily searches through 4GB of text in a snap.
4) People use apache.
This nicely combines with the points above.
Unlike a proper server like lighttpd whose CPU usage is undetectable (the crummy Via C7 was serving 100 HTTP hits/s and lighttpd used less than 1% CPU), apache will kill your box.
When the MySQL starts to die (it dies easily), clients start to hit F5 hard, and soon you have about 1000 apache processes, each holding a PHP interpreter, and each PHP interpreter holds an idle MySQL connection, waiting on a MyISAM lock, except one, which is doing some trivial UPDATE of your page view counter, but that takes some time, because the server is gone to lunch swapping, because of the 1000 apache and 1000 php and 1000 mysql processes.
Lighttpd uses no cpu for static pages. The only way for lighttpd to saturate your CPU is if you hit it hard with apachebench at like 20K requests/s. Then Lighttpd talks to a few, like 10 php-fcgi backends (2-4 per core is good) which talk to a few MySQL connections. Everything is a lot faster as a result, and when overloaded, it degrades gracefully, not explosively.
To get to the original question, you definitely want to profile your SQL queries. Add a query log to your PHP application which displays (only to you), the list of queries and the time they take, and also the time from the start of the PHP script to its end (header/footer includes are a good place for this).
For a complex page (excluding search) you'd expect about 3 ms MySQL and 3 ms PHP, that's a good target. You need a PHP compiled code cache of course.
For the current load, there are a couple things your can do. The most expensive, yet most detailed answers will be provided through a enterprise application such as "Gomez".
However, if you're looking to do this yourself, see my previous answers below or use shell utilities such as: htop, top, w, and utilize Apache server-status
Previous answers before question revision:
What you are asking for is sometimes called application profiling.
You need to create a rough memory formula like:
httpd ram + php memory usage + mysql process usage = total request memory footprint
You will also need a CPU formula, but you can also eyeball top during a load test.
Apache has the command 'ab'.
"ab is a tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server. It is designed to give you an impression of how your current Apache installation performs. This especially shows you how many requests per second your Apache installation is capable of serving." http://httpd.apache.org/docs/2.0/programs/ab.html
Here is a generic 'ab' benchmark command line:
ab -n 10 -c 1 http://www.yoursite.com/
# qty 10 total requests, 1 request at a time
The strategy is to test the per process (user) load on your application from the web page request through completion. If you can identify how much ram Apache, PHP, and MySQL uses for each request, then you can quickly identify your system capacity.
You'll probably have to use a mix of diagnostic tools like vmstat or top or iostat or ps, etc. to take a snapshot of what a number of requests will demand from your system.
Finally, you are going to want to install Xdebug. This tool will help you profile the php side of the application.
http://xdebug.org/
Here is IBM's tutorial on installing Xdebug:
http://www.ibm.com/developerworks/opensource/library/os-php-fastapps2/