My site has just recently run into some issues with High CPU usage from MYSQL. When I check the Parallels Power Panel, it says that this: "/usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/exa.example.com.pid --skip-external-locking" is using a high % of the CPU and the site becomes real slow.
I have checked every MySQL query on the entire site and there is nothing that should cause a long term problem like this. When I disable the routing file on the site, the load goes way down (almost 0%). As soon as it is turned back on, up to 100%. I came into work today to find that the server was down because of this.
The db has no more than 200,000 total records split up into 10 tables. I am starting to think that maybe the process(es) aren't finishing (hanging) or something else is wrong. You can load a page or two before the site comes to a stop.
What kind of ideas should I consider? I am somewhat of a newbie at configuring php and mysql so I haven't experimented with that too much yet.
UPDATE:
I've added this to the mysql config file:
log-slow-queries = log-slow-queries.log
long_query_time = 5
log-queries-not-using-indexes
Does that look okay? Of course, once I added that, the site seems to load a little better and the log file hasn't had anything written to it yet...
Consider enabling and then examining the slow query log, to help you find problematic queries.
Also you say that you "checked every MySQL query" - how? Did you run EXPLAIN plans for all of them? Sometimes queries can behave in ways that you don't expect.
Related
I've got a huge TTFB of around 6.5 seconds when the whole load time of my site is around 7 seconds
I've done the basics to try and reduce this - updating to the latest version of PHP, switched to https so I can enabled HTTP/2 where possible and enabling caches where possible such as OpCache which I've checked is up and running correctly with my phpinfo which you can see here
https://www.camp-site-finder.com/phpinfo.php
This and setting up caching plugins such as W3 Total Cache has reduces the issue but on search queries there still seems to a large wait
As you can see here for example if you check the Network tab of developer tools
https://www.camp-site-finder.com/?sfid=48&_sf_s=england
So my question really is how can I debug this, are there tools out there to test what is taking so long or is this a non issue? Is that wait period really acceptable? Any advice or pointing me in the direction of some research I could do I'd be hugely grateful.
If a search is slow, it is almost always a bottleneck of the database.
Which DB-Server do you use? I see MySQL and SQLight extensions active, but I guess it is the former. But do you use MySQL or MariaDB? You could try MariaDB or some other drop-in replacement for MySQL (like Percona) which should increase DB performance.
Also you should log slow queries in the DB server, so you can check which DB queries is that slow. I guess you might have to many joins. In that case you should need to restructure the database.
Additionally you could try do follow some basic db performance tips like these:
Indexing
Assigned memory
etc.
Just google for 'increase MySQL perfomance' and you should find plenty of adwise.
My server's load average shoots up to 150 (and the server is actually quite powerful 8cpus too much RAM etc), and the cause of that is MySQL taking all of my CPUs 700% !
I'm aware of Apache/MySQL tuning to meet better performance, I've done some, it worked a little bit but nowhere near the results I need.
All my problems are coming from this scenario: when website file based cache invalidates, PHP scripts run to remake those cached areas, generating MySQL queries (quite heavy queries, did some optimization on them too but they're still taxing on MySQL). That's quite normal, the problem is when a 100 people hit the website at that precise time when cache is invalid so they generate the same query 100 times - which makes MySQL sink all the way down with my server.
Are there any MySQL solutions to prevent the duplication of the same query? or is there any other technique to fix that special scenario?
I my opinion you would need another marker if rebuilding the cache is already in process, so that only one update query is done and until finished the old cache is used.
That would trigger the expensive query's just once.
For a better answer please explain how you invalidate your cache.
I'm trying to track down issues with an application [modx] I have several of these sites [about 10] on my server & was wondering how I can see what php is doing.
Pages on these sites are extremely slow while the same sites in dev are fine as are other php applications on the server.
I tried using xdebug to get an idea of what php was doing while processing these pages & where the bottleneck was occurring, but it only appeared to want to do anything on an error [there are no errors being thrown]
Any suggestions on how to track this down?
[linux/Centos5/php5.2.?/apache2]
Xdebug and webgrind are a nice way to see where your bottel necks are...
Read XDEBUG_PROFILE and Webgrind
Set up the php.ini to have xdebug profile your code on every run or if a special param is passed, then setup webgrind to read from the same directory xdebug writes its profile dumps to.
Webgrind will show you what functions and set of functions require the most time, it breaks it down and makes it easy to find slow and/or inefficient code. (eg. your script is calling "PDOStatement->execute" 300 times on a fast query [Or calling it once and a massively slow one] taking up 90% of the execution time).
The most commonly used tool, for finding bottlenecks in PHP, would be Xdebug. But you should also manually examine the codebase.
There are three different areas where you will have to focus on:
frontend performance
SQL queries
php logic itself
.. and the impact on the perceived speed is in this order.
You should start by running ySlow, and make sure that your site follows the guidelines as closely as possible.
The next step would be tracking down what SQL queries are executed, and (assuming you are using mysql) try to run them with EXPLAIN. Also, check the queries themselves. There might be some extremely stupid code there, like ORDER BY RAND() or use of LIKE in huge tables.
And the last stage would fixing it all would a hard looks at the code itself. Both on PHP and JavaScript side of things.
Also , you should upgrade to PHP 5.3, because your version is extremely outdated.
Usually when you don't know what you're looking for, you cannot spot it with tools like xdebug or other plugins/debug bars etc built into CMS/Framework, new relic is the simplest solution - you'll be able to spot bottlenecks after few min.
while new relic is a paid app, you can test if for free for first 14 days - it's more than enough to find problem.
It's great because it integrates all other tool's and data sources you usually use:
xdebug, cpu & i/o monitoring, mysql slowlog, queries log.
It will also show you if your app is slow on php/DB/frontend/network.
You should try it out instead of wasting time for debugging with other tools.
here is a guide for centos installation: https://newrelic.com/docs/php/php-agent-installation-redhat-and-centos
I am running wampserver on my windows vista machine. I have been doing this for a long time and it has been working great. I have completed loads of projects with this setup.
However, today, without me changing anything (no configuration etc) only PHP code changes, I find that every time I load pages of my site (those with user sessions or access the database) are really slow to load - Over 30 seconds, they use to take 1 or 2 seconds.
When I have a look at the task manager, I can see on page loads the httpd process jumps from 10mb to 30mb, 90mb, 120mb, 250mb and then back down again.
I have tested previous php code projects and they seem to all be slow as well!
What is going on?
Thanks all for any help on this confusion issue!
Check the following:
Check if you your data-access library to access your database has been changed/updated lately (if you use one).
Just a guess, but did you changed your antivirus/firewall (or settings) since last time you checked those previous projects? A more aggresive security can slow things a lot.
Did you changed the Apache/PHP/MySQL version in the WAMPSERVER menu?
Maybe you can try to reinstall WAMPSERVER (do this last and if it's not an hassle for you because I really doubt this will help but it can in some really really weird cases).
But from experience and the memory usage you explain in your question it seems that your SQL queries are long to execute and/or return a really large data set.
Try to optimise your queries, it can help for speed but not really memory usage (at least if the result set is the same). For the memory, maybe you can use LIMIT to reduce your returning data set (if your design allows it - but it should).
Since we don't really know what you do with your data, take note than "playing" (like parsing large XML documents) with large data sets can take much time/memory (again it depends much on what you do with all this data).
Bottom line, if nothing in this post helps, try to post more information on your setup and what exactly you do (with even code samples) when it's slow.
Try checking the size of your wamp log files.
i.e.
C:\wamp\logs
Sometimes, when they get really big, they can cause Apache to slow down.
Have you recently changed your network configuration or upgraded your system? That may be causing this issue through your network config or anti-virus/security software. People have had issues with zonealarm causing this in the past, for example.
Also, if you've recently switched from typing "127.0.0.1" to "localhost" or moved around networks, you may benefit from removing the IPV6 localhost setting from C:\Windows\System32\drivers\etc\hosts if you have one:
change the line with
::1 localhost
to
# ::1 localhost
I am surprised that no one has suggested this. You should always try to see why things are really slowing down:
Use Performance monitor to see where the bottleneck is, first. (perfmon.exe)
Is hard page fault actually the bottleneck? Is your hard drive busy reading/writing to the pagefile? Check the length of IO queue for the hard disk.
Are CPUs busy?
If nothing looks busy, use procmon to see if your php process is blocked on some system calls.
Hope this helps.
though not related to finding database bottlenecks, XDebug in combination with a cachegrind viewer (e.g. WebGrind, WinCacheGrind) can help you find the part of your PHP code, that takes longest to execute.
I've written a site CMS from scratch, and now that the site is slowly starting to get traffic (30-40k/day) Im seeing the server load a lot higher than it should be. It hovers around 6-9 all the time, on a quad core machine with 8gb of ram. I've written scripts that performed beautifully on 400-500k/day sites, so I'd like to think Im not totally incompetent.
I've reduce numbers of queries that are done on every page by nearly 60% by combining queries, eliminating some mysql calls completely, and replacing some sections of the site with static TXT files that are updated with php when necessary. All these changes affected the page execution time (index loads in 0.3s, instead of 1.7 as before).
There is virtually no IOwait, and the mysql DB is just 30mb. The site runs lighttpd, php 5.2.9, mysql 5.0.77
What can I do to get to the bottom of what exactly is causing the high load? I really wanna localize the problem, since "top" just tells me its mysql, which hovers between 50-95% CPU usage at all times.
Use EXPLAIN to help you optimize/troubleshoot your queries. It will show you how tables are referenced and how many rows are being read. It's very useful.
Also if you've made any modifications to your MySQL configuration, you may want to revisit that.
The best thing you can do is to profile your application code. Find out which calls are consuming so much of your resources. Here are some options (the first three Google hits for "php profiler"):
Xdebug
NuSphere PhpED
DBG
You might have some SQL queries that are very slow, but if they are run infrequently, they probably aren't a major cause of your performance problems. It may be that you have SQL queries that are more speedy, but they are run so often that their net impact to performance is greater. Profiling the application will help identify these.
The most general-purpose advice for improving application performance with respect to database usage is to identify data that changes infrequently, and put that data in a cache for speedier retrieval. It's up to you to identify what data would benefit from this the most, since it's very dependent on your application usage patterns.
As far as technology for caching, APC and memcached are options with good support in PHP.
You can also read through the MySQL optimization chapter carefully to identify any improvements that are relevant to your application.
Other good resources are MySQL Performance Blog, and the book "High Performance MySQL." If you're serious about running a MySQL-based website, you should be consulting these resources frequently.
mytop is a good place to start. It's basically top for MySQL, and will give you a window into what exactly your DB is doing:
http://jeremy.zawodny.com/mysql/mytop/
Noah
It could be any number of reasons, so it could take a lot of proding. A good first step would be to turn on the slow query log, and go over it by hand or with a parser. You can pick specific heavily used, slow queries to optimize (perhaps ones that hit something unindexed)