Recently we've been having problems with our LAMP setup and we started to see the number of MySQL database connections spike up every now and then. We suspect that some mysql operation is taking longer than usual and apache just started to build a backlog of connections to deal with incoming requests.
Question is, is there a way to per page statistics on things like average load time? median load time? max/min load time for each php page (page1.php, page2.php, page3.php etc). So that we can narrow down where the problem is. Is there such thing included as part of apache? Maybe a separate module?
From the log format, you can just log the time taken (%D) in your access logs, and after an incident, sort on time-taken, and check the urls. I'm not aware of any application that checks this out of the box, but a lot of applications can handle apache's access logs, so chances are there are those who can work with it. I seldomly look at page-specific logs, only server totals, so I can't help you there.
If MySQL is busy / the cause:
Close a connection to MySQL if you're done with it, so the connection is released sooner.
Increase the maximum allowed connections if you really need them.
If you still have hanging processes, check the output of SHOW FULL PROCESSLIST to see what queries are being performed.
You can enable the slow_query_log, logging all queries above a certain amount of miliseconds (in newer versions, old versions only supported seconds) or not using indexes. The command line tool mysqldumpslow can accurately group / count the queries.
If you have access to php.ini, you can use Xdebug : http://xdebug.org/
Related
I have an store running on Prestashop 1.5.4. I keep having a problem with the site behaviour.
Everytime i check this number
user#server:~$ mysql -se "show status like '%threads_connected%'"
If the number gets above 25, my site becomes really slow. Opening a page takes forever and page load can get as high as 1 - 2 minutes.
The only solution (temporary) is for me to restart the apache services.
I am pretty sure 25 is a pretty low number.
In case you need to know, i don't have direct access to my.ini nor to any access to MySQL Server Configuration. My database is stored in a shared (DBaSS).
I do however have full access to my webserver (apache2.conf, php.ini, www.example.com.conf)
A little info that might help:
-se "show variables like '%max%'"
Outputs:
I guess my question is how can i improve performance ? Can i improve this by limiting the number of "threads_connected" ?
footnote:
I am aware that there is probably a bad query somewhere in the code, however, at the moment i need a quick solution for this. As reviewing all queries can take some time.
EDIT #1
Perhaps this information might give some ideas
EDIT #2
Can i improve this by limiting the number of "threads_connected"?
Absolutely not.
All you could do by limiting the number of connections would be to cause errors for users of your site, because their particular Apache process wouldn't be able to connect to the database.
The problem is not the number of threads connected. That is a symptom of the real problem, which is that you have one or more queries that perform poorly, or that you do not have enough memory for Apache to scale up when traffic gets heavy, forcing your machine into heavily swapping, and thereby slowing down Apache to the point that it keeps connections open longer.
I need a quick solution for this
Sorry... but there isn't another solution other than to find the actual problem and fix it.
More useful than the number of connections is what are these connections doing right now?
SHOW FULL PROCESSLIST; in MySQL will answer that question. If they are just sleeping, they aren't hurting anything on the MySQL side, and you may want to limit the number of Apache processes, but if that is a side effect of the level of site traffic, then your server may actually be too small, or you may need to disable HTTP keepalive, or tweak the timeout, if browser connections are holding open Apache children that are idle, consuming memory.
I launched a website about a week ago and I sent out an email blast to a mailing list telling everyone the website was live. Right after that the website went down and the general error log was flooded with "exceeded process limit" errors. Since then, I've tried to really clean up a lot of the code and minimize database connections. I will still see that error about once a day in the error log. What could be causing this error? I tried to call the web host and they said it had something to do with my code but couldn't point me in any direction as to what was wrong with the code or which page was causing the error. Can anyone give me any more information? Like for instance, what is a process and how many processes should I have?
Wow. Big question.
Obviously, your maxing out your apache child worker processes. To get a rough idea of how many you can create, use top to get the rough memory footprint of one http process. If you are using wordpress or another cms, it could easily be 50-100m each (if you're using the php module for apache). Then, assuming the machine is only used for web serving, take your total memory, subtract a chunk for OS use, then divide that by 100m (in this example). Thats the max worker processes you can have. Set it in your httpd.conf. Once you do this and restart apache, monitor top and make sure you don't start swapping memory. If you do, you have set too high a number of workers.
If there is any other stuff running like mysql servers, make space for that before you compute number of workers you can have. If this number is small, to roughly quote a great man 'you are gonna need a bigger boat'. Just kidding. You might see really high memory usage for a http process like over 100m. You can tweak your the max requests per child lower to shorten the life of a http process. This could help clean up bloated http workers.
Another area to look at is time response time for a request... how long does each request take? For a quick check, use firebug plugin for firefox and look at the 'net' tab to see how long it takes for your initial request to respond back (not images and such). If for some reason request are taking more than 1 or 2 seconds to respond, that's a big problem as you get sort of a log jam. The cause of this could be php code, or mysql queries taking too long to respond. To address this, make sure if you're using wordpress to use some good caching plugin to lower the stress on mysql.
Honestly, though, unless your just not utilizing memory by having too few workers, optimizing your apache isn't something easily addressed in a short post without detail on your server (memory, cpu count, etc..) and your httpd.conf settings.
Note: if you don't have server access you'll have a hard time figuring out memory usage.
The process limit is typically something enforced by shared webhost providers, and generally has to do with the number of processes executing under your account. This will typically equate to the number of connections made to your server at once (assuming one PHP process per each connection).
There are many factors that come into play. You should figure out what that limit is from your hosting provider, and then find a new one that can handle your load.
My linux server websites keep going down again and again but SSH, FTP, etc are alive. So I had a look at the server through SSH and used top command which lists all the processes. It shows that when some PHP pages are executed, mysql CPU usage reaches 100%. So is there any command/log which can be used to find out which PHP pages are taking up so much of mysql usage? Thank you...
You may want to take a look at your Apache log format to see if it includes the %D parameter as this indicates the amount of time taken to to serve a request in microseconds.
If you exclude anything but requests to PHP scripts, you should get an idea of which scripts are taking the longest suggesting high execution time. Obviously this could also mean a very large response payload...
There are multiple aspects to resource consumption.
As mobius mentioned, you can use SHOW FULL PROCESSLIST in MySQL to see what is currently running. Look at the processed taking longer than you would expect and check out the query to find hints about where it originates in your application.
The problem may not be with the application. It might simply be a matter of tuning MySQL, which will be about adding or changing indexes most of the time. EXPLAIN is the command that will you help analyze the execution plan MySQL decided to use. Reading EXPLAIN takes some practice. The best reference I have is High Performance MySQL.
You can also use the MySQL slow query log to get information about the slow queries happening when you are not in front of the server.
If MySQL is running at 100%, you will probably find the problem from there. If you really want to track the usage from PHP, you can set up XHProf, a high performance profiler created by Facebook to run on production sites. You can set it up to sample one request out of 100 and get a bigger picture of the performance of your site. There are a few articles out there that explain how to set it up.
Finally, XDebug and KCacheGrind can be used in development to profile one request at a time.
If MySQL is getting stuck at 100% then you've probably got some badly tuned MySQL queries inside one of your PHP applications. This time will clock up in the MySQL daemon and so won't show up in the %D value. This could be indexes out of date.
If you have access to the D/B through at the command prompt through SSH then you could try doing an ANALYZE TABLE and OPTIMIZE TABLE on any large tables. Also look at "The Slow Query Log" in the MySQL documentation.
Unfortunately fixing this will probably need you to get into the Application internals.
mytop - http://jeremy.zawodny.com/mysql/mytop/ (SHOW FULL PROCESSLIST on your mySQL)
Xdebug Profiler - http://xdebug.org/docs/profiler
We have an application that is comprised of a couple of off the shelf PHP applications (ExpressionEngine and XCart) as well as our own custom code.
I did not do the actual analysis so I don't know precisely how it was determined, but am not surprised to hear that too many MySQL connections are being left unclosed (I am not surprised because I have been seeing significant memory leakage on our dev server, where over the course of a day or two, starting from 100MB upon initial boot, the entire gig of ram gets consumed, and very little of it is cached).
So, how do we go about determining precisely which PHP code is the culprit? I've got prior experience with XDebug, and have suggested that, when we've gotten our separate, staging environment reasonably stable, that we retrofit XDebug on dev and use that to do some analysis. Is this reasonable, and/or does anybody else have more specific and/or additional suggestions?
You can use the
SHOW PROCESSLIST
SQL command to see what processes are running. That will tell you the username, host, database, etc that are in use by each process. That should give you some idea what's going on, especially if you have a number of databases being accessed.
More here: https://dev.mysql.com/doc/refman/8.0/en/show-processlist.html
This should not be caused by a php code because mysql connections are supposed to be automatically closed.
cf : http://www.php.net/manual/function.mysql-connect.php :
The link to the server will be closed
as soon as the execution of the script
ends, unless it's closed earlier by
explicitly calling mysql_close().
Some suggestions :
does your developper has technically a direct access to your production mysql server ? if yes, then they probably just leave their Mysql Manager open :)
do you have some daily batch process ? if yes, maybe that there are some zombi process in memory
PHP automatically closes any mysql connections when the page ends. the only reason that a PHP web application would have too many unclosed mysql connections is either 1) you're using connection pooling, or 2) there's a bug in the mysql server or the connector.
but if you really want to look at your code to find where it's connecting, see http://xdebug.org/docs/profiler
As others said, PHP terminates MySQL connections created through mysql_connect or the msqli/PDO equivalents.
However, you can create persistent connections with mysql_pconnect. It will look for existing connections open and use those; if it can't find one, it will open a new one. If you had a lot of requests at once, it could have caused loads of connections to open and stay open.
You could lower the maximum number of connections, or lower the timeout for persistent connections. See the comments at the bottom of the man page for more details.
I used to run a script that polled SHOW STATUS for thread count and I noticed that using mysql_pconnect always encouraged high numbers of threads. I found that very disconcerting because then I couldn't tell when my connection rate was actually dropping. So I made sure to centralize all the places where mysql_connect() was called and eliminate mysql_pconnect().
The next thing I did was look at the connection timeouts and adjust them to more like 30 seconds because. So I adjusted my my.cnf with
connect-timeout=30
so I could actually see the number of connections drop off. To determine the number of connections you need open is dependent on how many apache workers you're running times the number of database connections they each will open.
The other thing I started doing was adding a note to my queries in order to spot them in SHOW PROCESSLIST or mytop, I would add a note column to my results like:
$q = "SELECT '".__FILE__.'.'.__LINE__."' as _info, * FROM table ...";
This would show me the file issuing the query when I looked at mytop, and it didn't foil the MySQL query cache like using
/* __FILE__.'.'.__LINE__ */
at the start of my query would.
I suppose another couple of things I can do, with regard to the general memory issue, as opposed specifically to MySQL, and particularly within the context of our own custom code, would be to wrap our code with calls to one or the other of the following PHP built-in functions:
memory_get_usage
memory_get_peak_usage
In particular since I am currently working on logging from some custom code, I can log the memory usage while I'm at it
I'm trying to debug an error I got on a production server. Sometimes MySQL gives up and my web app can't connect to the database (I'm getting the "too many connections" error). The server has a few thousand visitors a day and on the night I'm running a few cron jobs which sometimes does some heavy mysql work (Looping through 50 000 rows, inserting and deletes duplicates etc)
The server runs both apache and mysql on the same machine
MySQL has a pretty standard based configuration (max connections)
The web app is using PHP
How do I debug this issue? Which log files should I read? How do I find the "evil" script? The strange this is that if I restart the MySQL server it starts working again.
Edit:
Different apps/scripts is using different connectors to its database (mostly mysqli but also Zend_Db)
First, use innotop (Google for it) to monitor your connections. It's mostly geared to InnoDB statistics, but it can bet set to show all connections including those not in a transaction.
Otherwise, the following are helpful: Use persistent connections / connection pools in your web apps. Increase your max connections.
It's not necessarily a long-running SQL query.
If you open a connection at the start of a page, it won't be released until the PHP script terminates - even if there is no query running.
You should add some stats to your pages to find out the slowest ones, and the most-hit ones. Closing the connection early would help, if possible.
Try using persistent connections (mysql_pconnect), it will help reduce the server load caused by constantly opening and closing MySQL connections.
The starting point is probably to use mysqladmin processlist to get a list of the processes on the mysql server. The next step depends on what you find.