Online project running on a period of time, notify the monitoring can see, every night a MySQL connection number is particularly high, the project development using PHP, I don't know how to control the number of connections, after all, it doesn't like memory resident type language that can realizes the connection pool at the code level.
By default 151 is the maximum permitted number of simultaneous client connections in MySQL 5.5. If you reach the limit of max_connections you will get the “Too many connections” error when you to try to connect to your MySQL server – which means all available connections are in use by other clients.
MySQL permits one extra connection on top of the max_connections limit which is reserved for the database user having SUPER privilege in order to diagnose connection problems. Normally the administrator user has this SUPER privilege. You should avoid granting SUPER privilege to app users.
MySQL uses one thread per client connection and many active threads are performance killer. Usually a high number of concurrent connections executing queries in parallel can cause significant slowdown and increase chances for deadlocks. Prior to MySQL 5.5, it doesn’t scale well although MySQL is getting better and better since then – but still if you have hundreds of active connections doing real work (this doesn’t count sleeping [idle] connections) then the memory usage will grow. Each connection requires per thread buffers. Also implicit in memory tables require more memory plus memory requirement for global buffers. On top of that, tmp_table_size/max_heap_table_size that each connection may use, although they are not allocated immediately per new connection.
Most of the time, an overly high number of connections is the result of either bugs in applications not closing connections properly or because of wrong design, like the connection to mysql is established, but then the application is busy doing something else before closing MySQL handler. In cases where an application doesn’t close connections properly, wait_timeout is an important parameter to tune and discard unused or idle connections to minimize the number of active connections to your MySQL server – and this will ultimately help to avoid the “Too many connections” error. Although some systems are running alright with even a high number of connected threads, most of the connections are idle. In general sleeping threads do not take too much memory – 512 KB or less. Threads_running is a valuable metric to monitor as it doesn’t count sleeping threads – it shows active and the amount of queries currently processing, while threads_connected status variables show all connected threads value including idle connections as well.
If you are using connection pool on the application side, max_connections should be bigger than max connections on the pool side. Connection pooling is also a good alternative if you are expecting a high number of connections. Now what should be the recommended value for max_connections ?
There is no single right answer for that question. It depends on the amount of RAM available and memory usage for each connection. Increasing max_connections value increases the number of file descriptors that mysqld requires. Note: there is no hard limit to setting up maximum max_connections value. So, you have to choose max_connections wisely as per your workload, number of simultaneous connections to MySQL server etc. In general allowing too high of a max_connections value is not recommended because in case of some locking conditions or slowdowns if all those connections running huge contention issue may raise. In case of active connections using temporary/memory tables memory usage can go even higher. On systems with small RAM or with hard number of connections control on the application side we can use small max_connections values like 100-300. Systems with 16G RAM or higher max_connections=1000 is a good idea, of course per-connection buffer should have good/default values while on some systems we can see up to 8k max connections, but such systems usually became down in case of load spikes.
To deal with it, Oracle and the MariaDB team implemented thread pool. With a properly configured thread pool you may expect throughput to NOT decrease for up to few thousand concurrent connections, for some types of workload at least.
NOTE: Beware, that in MySQL 5.6 a lot of memory is allocated when you set max_connections value too high.
Related
I have a large PHP/MySQL application where infrequently it will break with the error:
"already has more than max_user_connections active connections in xyz.php"
The max_user_connections limit is already set to 100 but I would prefer to see if there is an issue with my application, such as open connections or bad connections that is causing this.
I've monitored the active processes in PHPMyAdmin and can not see any that particularly hang or seem an issue.
Is there any suggestions on how to debug my code or find any possible causes to this? Would it be specifically in the xyz.php file mentioned, or is this just because my mysql connection classes are in there? The error is so temporary, only for seconds, I'm at a loss on how to hunt the cause down.
Connections beyond max_user_connections configuration in mysql always leading to mysql reject new connections.For dealing with this situation
You could use a connection pool to manage connections with mysql,which connection is reuseable.If you do not want to use a pool, remmbering closing connection properly!
Turning up max_user_connections in my.cnf,you should measuring your host's capacity for a good configuration.Nerver set a infinite or unlimited value. Overly high number of connections will run out of host's resources,and make poor performance query because race condition.
I am having very high CPU spikes on mysqld process (greater than 100%, and even saw a 300% at one point). My load average is around: .25, .34, .28.
I read this great post about this issue: MySQL high CPU usage
One of the main things to do is disable persistent connections. So I checked my php.ini and mysql.allow_persistent = on and mysql.max_persistent = -1 -- which means no limit.
This raises a few questions for me before changing anything just to be sure:
If my mysqld process is spiking over 100% every couple seconds shouldn't my load average be higher then they are?
What will disabling persistent links do - will my scripts continue to function as is?
If I turn this off and reload php what does this mean for my current users as there will be many active users.
EDIT:
CPU Info: Core2Quad q9400 2.6 Ghz
Persistent connections won't use any CPU by themselves - if nothing's using a connection, it's just sitting idle and only consumes a bit of memory and occupies a socket.
Load averages are just that - averages. If you have a process that alternates between 0% and 100% 10 times a second, you'd get a load average of 0.5. They're good for figuring out long-term persistent high cpu, but by their nature hide/obliterate signs of spikes.
Persistent connections with mysql are usually not needed. MySQL has a relatively fast connection protocol and any time savings from using persistent connections is fairly minimal. The downside is that once a connection goes persistent, it can be left in an inconsistent state. e.g. If an app using the connection dies unexpectedly, MySQL will not see that and start cleaning up. This means that any server-side variables created by the app, any locks, any transactions, etc... will be left at the state they were in when the app crashed.
When the connection gets re-used by another app, you'll start out with what amounts to dirty dishes in the sink and an unflushed toilet. It can quite easily cause deadlocks because of the dangling transactions/locks - the new app won't know about them, and the old app is no longer around to relinquish those.
Spikes are fine. This is MySQL doing work. Your load average seems appropriate.
Disabling persistent links simply means that the scripts cannot use an existing connection to the database. I wouldn't recommend disabling this. At the very least, if you want to disable them, do it on the application later, rather than on MySQL. This might even increase load slightly, depending on the conditions.
Finally, DB persistence has nothing to do with the users on your site (generally). Users make a request, and once all of the page resources are loaded, that is it, until the next request. (Except in a few specific cases.) In any case, while the request is happening, the script will still be connected to the DB.
In the Java world it is pretty standard for app servers to pool "expensive" resources, like DB connections. On the other hand in dynamic languages, most stacks have little to do with pooled resources and especially DB connections.
E.g. for the popular PHP+MySQL combo, I've rarely seen it used with persistent connection, which can be considered poor-mans pooled connections.
If the concept of pooling DB connections is not that widely implemented, does this mean that the performance/scalability gains might not be all that important, in real-life deployments?
The main reason for connection pooling is the overhead in establishing the connection in the first instance. I have seen this take up to 0.5 seconds in the past.
In a high transactional environment, the ability to keep a connection open, and send multiple requests down the connection, one after the other have significant savings. Therefore, you may not see the gains in a low transactional database, but your application is not going to scale as well, if you ignore this useful pattern.
It also helps to managed the number of open connections in a much clearer way.
If the concept of pooling DB
connections is not that widely
implemented, does this mean that the
performance/scalability gains might
not be all that important, in
real-life deployments?
It just means that there's no built-in connection pool for PHP. Doesn't mean you can't use one, though, for instance with Postgres you can use PGPool.
My general opinion is that connection pools generally aren't used in environments where speed isn't considered as important as other concerns. AOLServer, for example, uses a dynamic language (Tcl) but is highly performant and uses connection pools.
Connection polling is often used because some database vendors limit the number of connections that you have to a given database depending on your license. Open Source databases do not have such limits because they are free. So it is not much of a problem for MySQL.
Another reason to use connection polling is to limit the number of current connections to the database server, as each new connection consumes a lot of memory and you do not want to exhaust your servers memory.
The problem with persistent connections is that they never close until the client processes die. Client processes are in reality the Web server processes handling PHP requests. So, if you configure your Web server to limit the number of simultaneous requests, you also limit the number of opened persistent database connections. You can do that in Apache setting the MaxClients parameter to a reasonable value that does not exhaust your server RAM.
BTW, it would also be wise to move all your static content (CSS, JavaScript, images, etc..) to a separate multithreaded Web server (Nginx, lighttpd, etc..) so the simultaneous user accesses do not make your Apache fork to much processes.
I am using MySQL and PHP with 2 application servers and 1 database server.
With the increase of the number of users (around 1000 by now), I'm getting the following error :
SQLSTATE[08004] [1040] Too many connections
The parameter max_connections is set to 1000 in my.cnf and mysql.max_persistent is set to -1 in php.ini.
There are at most 1500 apache processes running at a time since the MaxClients apache parameter is equal to 750 and we have 2 application servers.
Should I raise the max_connections to 1500 as indicated here?
Or should I set mysql.max_persistent to 750 (we use PDO with persistent connections for performance reasons since the database server is not the same as the application servers)?
Or should I try something else?
Thanks in advance!
I think your connections aren't closing fast enough and they stack until the default time has reached. I had same problem and with wait_timeout I solved things out.
You can try to setup in my.cnf
set-variable = max_connections=1000 // max connection
set-variable = max_user_connections=100 // max user connection per hour
wait_timeout = 60 // wait timeout for connection in seconds
as will terminate any existing connections after 60 seconds has passed
I think you should check out the PHP code if you can get rid of the persistent connections.
The problem with persistent connections is that the php instance keeps them open even after script exits until the data has been sent to the client and php instance is freed to next customer.
Another problem with persistent connections is that some PHP code might leave the socket with different settings than in the startup, with different locales or with temporary tables.
If you can rewrite the code that for every connection theres only one or few mysql_connects and the database handle is passed to different parts of the code or kept in GLOBAL the performance impact of losing persistent connections is negligible.
And, of course, there's little harm in doubling the max_connections. It's not very useful with PHP anyway as PHP/Apache child exits quite often anyway and closes the handles. The max_connections is more useful in other environments.
Trying to separate out my LAMP application into two servers, one for php and one for mysql. So far the application connects locally through a file socket and works fine.
I'm worried about the number connections I can establish if it is over the network. I have been testing tcp connections on unix for benchmark purposes and I know that you cannot exceed a certain amount of connections per second otherwise it halts due to the lack of resources (be it sockets, or file handles or whatever). I also understand that php does not implement connection pooling so for each page load a new connection over the network must be made. I also looked into pconnect for php and it seems to bring more problems.
I know this is a very very common setup (php+mysql), can anyone provide some typical usage and statistics they get out of their servers? Thanks!
The problem is not related to running out of connections allowed my MySQL. The main problem is that unix cannot very quickly create and tear down tcp connections. Sockets end up in TIME_WAIT and you have to wait for a period before you free up more sockets to connect again. These two screenshots clearly shows this pattern. MySQL does work up to a certain point and then pauses because the web server ran out of sockets. After certain amount of time passed, the web server was able to make new connections.
alt text http://img35.imageshack.us/img35/3809/picture4k.png
alt text http://img35.imageshack.us/img35/4580/picture2uyw.png
I think the limit is at 65535. So you'd have to have 65535 connections at the same time to hit that limit since a regular mysql connection closes automatically.
mysql_connect()
Note: The link to the server will be closed as soon as the execution of the script ends, unless it's closed earlier by explicitly calling mysql_close().
But if you're using a persistent mysql connection, then you can run into trouble.
Using persistent connections can require a bit of tuning of your Apache and MySQL configurations to ensure that you do not exceed the number of connections allowed by MySQL.
Each MySQL connection actually uses several meg of ram for various buffers, and takes a while to set up, which is why MySQL is limited to 100 concurrent open connections by default. You can up that limit, but it's better to spend your time trying to limit concurrent connections, via various methods.
Beware of raising the connection limit too high, as you can run out of memory (which, I believe, crashes mysql), or you may push important things out of memory. e.g. MySQL's performance is highly dependent on the OS automatically caching the data it reads from disk in memory; if you set your connection limit too high, you'll be contending for memory with the cache.
If you don't up your connection limit, you'll run out of connections long before your run out of sockets/file handles/etc. If you do increase your connection limit, you'll run out of RAM long before you run out of sockets/file handles/etc.
Regarding limiting concurrent connections:
Use a connection pooling solution. You're right, there isn't one built in to PHP, but there are plenty of standalone ones out there to choose from. This saves expensive connection setup/tear down time.
Only open database connections when you absolutely need them. In my current project, we automatically open a database connection when the first query is issued, and not a moment before; we also release the connection after we've done all our database work, but before the page's HTML is actually generated. The shorter the period of time you hold connections open, the fewer connections will be open simultaneously.
Cache what you can in a lighter-weight solution like memcached. My current project temporarily caches pages displayed to anonymous users (since every anonymous user gets the same HTML, in the end -- why bother running the same database queries all over again a few scant milliseconds later?), meaning no database connection is necessary at all. This is especially useful for bursts of anonymous traffic, like a front-page digg.