This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
MySQL server has gone away - in exactly 60 seconds
We're using the mysql_connect method in PHP to create database handles for a daemon. The problem is, that connection might not be used for more than 8 hours (sometimes for more than a few weeks.)
We're running into issues where MySQL will end the session because the wait_timeout is reached.
http://dev.mysql.com/doc/refman/5.0/en/gone-away.html
We don't want to increase this value for ALL connections. Is there a way to increase the timeout for that handle only through PHP? Through MySQL?
It's not good to hang onto DB connections for long periods because the DB only provides a fixed number of connections at any one time; if you're using one up for ages it means your DB has less capacity to deal with other requests, even if you're not actually doing anything with that connection.
I suggest dropping the connection if the program is finished using it for the time being, and re-connect when the time comes to do more DB work.
In addition, this solution will protect your program from possible database downtime, ie if you need to reboot your DB server (it happens, even in the best supported network). If you keep the connection alive (ie a DB ping as per the other answers), then an event like that will leave you with exactly the same problem you have now. With a properly managed connection that is dropped when not needed, you can safely leave your daemon running even if you have planned downtime on your DB; as long as it remains idle for the duration, it needn't be any the wiser.
(as an aside, I'd also question the wisdom of writing a PHP program that runs continuously; PHP is designed for short duration web requested. It may be capable of running long term daemon programs, but there are better tools for the job)
For this problem, the best way is use mysql_ping();
Check it out on manual PHP mysql_ping()
Related
I'm tuning my project MYSQL database, as many people I got suggestion to reduce
wait_timeout
, but it is unclear for me, does this session variable exclude query execution time, or it includes it? I have set it for 5 seconds, taking into account that I may have queries which are being executed for 3-5 seconds sometimes (yea that is slow there are few of them, but they still exist), so mysql connections have at least 1-2 seconds to be taken by PHP scripts before they are closed by MYSQL.
In MySQL docs there is no clear explanation about how it starts counting that timeout and if it includes execution time. Perhaps your experience may help. Thanks.
Does the wait_timeout session variable exclude query execution time?
Yes, it excludes query time.
max_execution_time controls how long the server will keep a long-running query alive before stopping it.
Do you use php connection pools? If so 5 seconds is an extremely short wait_time. Make it longer. 60 seconds might be good. Why? the whole point of connection pools is to hold some idle connections open from php to MySQL, so php can handle requests from users without the overhead of opening connections.
Here's how it works.
php sits there listening for incoming requests.
A request arrives, and the php script starts running.
The php script asks for a database connection.
php (the mysqli or PDO module) looks to see whether it has an idle connection waiting in the connection pool. If so it passes the connection to the php script to use.
If there's no idle connection php creates one and passes it to the php script to use. This connection creation starts the wait_timeout countdown.
The php script uses the connection for a query. This stops the wait_timeout countdown, and starts the max_execution_time countdown.
The query completes. This stops the max_execution_time countdown and restarts the wait_timeout countdown. Repeat 6 and 7 as often as needed.
The php script releases the connection, and php inserts it into the connection pool. Go back to step 1. The wait_time is now counting down for that connection while it is in the pool.
If the connection's wait_time expires, php removes it from the connection pool.
If step 9 happens a lot, then step 5 also must happen a lot and php will respond more slowly to requests. You can make step 9 happen less often by increasing wait_timeout.
(Note: this is simplified: there's also provision for a maximum number of connections in the connection pool.)
MySQL also has an interactive_timeout variable. It's like wait_timeout but used for interactive sessions via the mysql command line program.
What happens when a web user makes a request and then abandons it before completion? For example, a user might stop waiting for a report and go to another page. In some cases, the host language processor detects the closing of the user connection, kills the MySQL query, and returns the connection to the pool. In other cases the query either completes or hits the max_execution_timeout barrier. Then the connection is returned to the pool. In all cases the wait_timeout countdown only is active when a connection is open but has no query active on it.
A MySQL server timeout can occur for many reasons, but happens most often when a command is sent to MySQL over a closed connection. The connection could have been closed by the MySQL server because of an idle-timeout; however, in most cases it is caused by either an application bug, a network timeout issue (on a firewall, router, etc.), or due to the MySQL server restarting.
It is clear from the documentation that it does not include the query execution time. It is basically the maximum idle-time allowed between two activities. If it crosses that limit, server closes the connection automatically.
The number of seconds the server waits for activity on a
noninteractive connection before closing it.
From https://www.digitalocean.com/community/questions/how-to-set-no-timeout-to-mysql :
Configure the wait_timeout to be slightly longer than the application connection pool's expected connection lifetime. This is a good safety check.
Consider changing the waittimeout value online. This does not require a MySQL restart, and the waittimeout can be adjusted in the running server without incurring downtime. You would issue set global waittimeout=60 and any new sessions created would inherit this value. Be sure to preserve the setting in my.cnf. Any existing connections will need to hit the old value of waittimeout if the application abandoned the connection. If you do have reporting jobs that will do longer local processing while in a transaction, you might consider having such jobs issue set session wait_timeout=3600 upon connecting.
From the reference manual,
Time in seconds that the server waits for a connection to become active before closing it.
In elementary terms, How many SECONDS will you tolerate someone reserving your resources and doing nothing? It is likely your 'think time' before you decide which action to take is frequently more than 5 seconds. Be liberal. For a web application, some processes take more than 5 seconds to run a query and you are causing them to be terminated at 5 seconds. The default is 28800 seconds which is not reasonable. 60 seconds could be a reasonable time to expect any web based process to be completed. If your application is also used in a traditional workplace environment, a 15 minute break is not unreasonable. be liberal to avoid 'bad feedback'.
I'm having trouble understanding an example of when this function would be used, and how it would be implemented. Could anyone provide some clarity on the subject? The php manual provides this information but I'd really appreciate it if someone could break it down "barney style" for me.
Thanks in advance.
Checks whether the connection to the server is working. If it has gone down, and global option mysqli.reconnect is enabled an automatic reconnection is attempted.
This function can be used by clients that remain idle for a long while, to check whether the server has closed the connection and reconnect if necessary.
http://php.net/manual/en/mysqli.ping.php
Lets say you have an PHP-Job that is running from an crontab under linux.
And the script maybe takes a long time to run.
Plus the script is runnnig more than one time at the same time.
Whitin the script you connect to your DB at the beginning, then the script does a lot of work (maybe download large data, prepare large data ....) and it is here and there using the database. But in same cases the database connection is lost because of too long idle time (Database Configuration). Some script maybe need 1 min. to download and another istance needs more than 5 hours.
Here comes the mysqli_ping function and handles that. Instead of allways reconnect to the database (before each query, to be really really sure its connected) the mysql_ping can test the connection if still working. if not you can then reconnect the connection.
Topic here: max_connction_timeout, max_allowed_connection, max_idle_time
see MYSQL Pages
Kindly, Barney
if you have a long-running script, for an example some back-end process such as cron job where there could be a time span between connecting and applying queries, mysqli_ping comes handy in checking db connection availability.
I'm having a problem that I hope someone can help me out with.
Currently, every now and again we receive an error when our scripts (Java and PHP) try to connect to the localhost mysql database.
Host 'myhost' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'.
This issue appears to mainly occur in the early hours of the morning. After alot of searching to figure out why this may be occuring I have finally come to the conclusion that it may be due to the fact our hosting company runs their backup processes around this time. My theory is that during this backup process (this is also our busiest period) we end up using up all our connections and so this error occurs.
I have talked to our hosts about changing the times these backups occur but they have stated that this is not possible and that is simply the times the backups start to ensure they are finished in time for the day (Even though we have informed them our critical period is at the precise times the backups occur).
The things I have connecting to the server are:
PHP website
PHP files run using chron jobs
A couple of java applications to run as socket listeners that listen for incoming port connections and uses the mysql database for checking user credentials and checking outstanding messages.
We typically have anywhere from 300 - 600 socket connections open at any one time and the average activity on these are about 1-3 request per second.
I have also installed monit and munin with some mysql plugins on the server in the hope they may help auto resolve this issue however these do not see to resolve the issue.
My questions are:
Is there something I can do to auto poll the mysql database so if this occurs I can auto flush the database to clear
Is this potentially even related to the server backup. It seems a coincidence it happens 95% of the time during the period the backups occur.
Any other ideas that may help. Links to other websites, or questions I could put to our host to help out.
We are currently running on a PHP Version 5.2.6-1+lenny9 server with Apache.
If any more information is required to help, please let me know. Thanks.
UPDATE:
I am operating on a shared virtual host and am pretty sure I close my website connections as I have this code in my database class
function __destruct() {
#mysql_close($this->link);
}
I'm pretty sure I'm not using persistant connections via my PHP script as I connect to the db the #mysql_connect command.
UPDATE:
So I changed the max_connections limit from 100 - 200 and I changed the mysql.persistant variable from On to Off in php.ini. Now for two nights running the server has gone done and mainly the connection to the mySql database. I have one 1GB of RAM on the server but it never seems to get close to that. Also looking at my munin logs the connections never seem to hit the 200 mark and yet I get errors in my log files something like
SQLException: Too many connections
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
SQLException: null, message from server: "Can't create a new thread (errno 12); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug.
SQLState :: SQLException: HY000, VendorError :: SQLException: 1135
We've had a similar problem with out large ecommerce installation using MySQL as a backend. I'd suggest you alter the "max_connections" setting of the MySQL instance, then (if necessary) alter the number of file descriptors using "ulimit" before starting MySQL (we use "ulimit -n 32768" in /etc/init.d/mysql).
It's been suggestion I post an answer to this question although I never really got it sorted.
In the end I ended up implementing a Java connection pooling class which enabled me to share connections whilst maintaining a upper limit on the number of max connections I wanted. It was also suggested I increase the RAM and increase the number of max connections. I did both these things although they were just bandaids to the problem. We also ended up moving hosting providers as the ones we were with were not very co-ooperative.
After these minor implementations I haven't noticed this issue occur for at least 8 months which is good enough for me.
Other suggestions over time have to also implement a Thread pooling facility, however current demand does not require this need.
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
close mysql connection important?
How important is it close an mysql connection in a website and why?
Every database engine has a limit on a maximum number of simultaneous connections. So if you don't close the connection, mysql can run out of available connections (default max_connections is 100). In addition, each connection you hold consumes server's resources (memory, a thread to listen, might be open filehandles).
CAVEAT
This does NOT hold true if the only things opening connection are web apps from ONE server and they use pooled connections. In such a case, you don't risk opening more and more new connections (since every time your app needs a new one, it picks the ones available from the pool); and closing and re-opening the pool's connections just wastes resources.
I don't know about other languages, but php closes the connection automatically on the end of script execution.
In general case, you close connection so that sql server doesn't have to wait for more commands from the website (which it won't receive, because the script has finished execution), and doesn't hit connection quota (if it has any).
PHP will automatically close the connection when the script exits, so I wouldn't normally worry about it too much.
The database server will have a finite possible number of simultaneous connections, so on a very heavily loaded site it might help to free the connection as soon as possible.
If you don't properly close your connections you could get into trouble. You will probably receive Too many connections errors. See here
Like so often, it depends.
What could happen:
The connection could stay open and you can run out of the maximum open connections
Your changes aren't committed
Depending on program language, middle ware, ...
Because there is a limit to the number of sockets connections that can be opened.
I'm not THE programmer, but I think it's better to close it as soon as you don't need it. Say, for example, you open the connection, execute the query and for some reason, working with that data takes a long time. If you haven't closed the connection yet (and the program/script hasn't ended either, so it hasn't automatically closed the connection -if there's such a chance), there will be a busy resource which is doing nothing in MySQL. A connection is open but you're not using it, that means another client using the service probably will see the great "Error: Can't connect to the DB" message.
So, in my humble opinion, it's better to connect to the DB, gather the requiered data, close the connection and then process the data. That at least will leave one available connection to future clients (which is really important if you have high-concurrency apps).
Nevertheless, that kind of behaviour depends on the programming language but all I know can retrieve data to a variable and it's connection-independent, so I can just close it and keep working with the data while leaving the connection available to another client.
We have an application that is comprised of a couple of off the shelf PHP applications (ExpressionEngine and XCart) as well as our own custom code.
I did not do the actual analysis so I don't know precisely how it was determined, but am not surprised to hear that too many MySQL connections are being left unclosed (I am not surprised because I have been seeing significant memory leakage on our dev server, where over the course of a day or two, starting from 100MB upon initial boot, the entire gig of ram gets consumed, and very little of it is cached).
So, how do we go about determining precisely which PHP code is the culprit? I've got prior experience with XDebug, and have suggested that, when we've gotten our separate, staging environment reasonably stable, that we retrofit XDebug on dev and use that to do some analysis. Is this reasonable, and/or does anybody else have more specific and/or additional suggestions?
You can use the
SHOW PROCESSLIST
SQL command to see what processes are running. That will tell you the username, host, database, etc that are in use by each process. That should give you some idea what's going on, especially if you have a number of databases being accessed.
More here: https://dev.mysql.com/doc/refman/8.0/en/show-processlist.html
This should not be caused by a php code because mysql connections are supposed to be automatically closed.
cf : http://www.php.net/manual/function.mysql-connect.php :
The link to the server will be closed
as soon as the execution of the script
ends, unless it's closed earlier by
explicitly calling mysql_close().
Some suggestions :
does your developper has technically a direct access to your production mysql server ? if yes, then they probably just leave their Mysql Manager open :)
do you have some daily batch process ? if yes, maybe that there are some zombi process in memory
PHP automatically closes any mysql connections when the page ends. the only reason that a PHP web application would have too many unclosed mysql connections is either 1) you're using connection pooling, or 2) there's a bug in the mysql server or the connector.
but if you really want to look at your code to find where it's connecting, see http://xdebug.org/docs/profiler
As others said, PHP terminates MySQL connections created through mysql_connect or the msqli/PDO equivalents.
However, you can create persistent connections with mysql_pconnect. It will look for existing connections open and use those; if it can't find one, it will open a new one. If you had a lot of requests at once, it could have caused loads of connections to open and stay open.
You could lower the maximum number of connections, or lower the timeout for persistent connections. See the comments at the bottom of the man page for more details.
I used to run a script that polled SHOW STATUS for thread count and I noticed that using mysql_pconnect always encouraged high numbers of threads. I found that very disconcerting because then I couldn't tell when my connection rate was actually dropping. So I made sure to centralize all the places where mysql_connect() was called and eliminate mysql_pconnect().
The next thing I did was look at the connection timeouts and adjust them to more like 30 seconds because. So I adjusted my my.cnf with
connect-timeout=30
so I could actually see the number of connections drop off. To determine the number of connections you need open is dependent on how many apache workers you're running times the number of database connections they each will open.
The other thing I started doing was adding a note to my queries in order to spot them in SHOW PROCESSLIST or mytop, I would add a note column to my results like:
$q = "SELECT '".__FILE__.'.'.__LINE__."' as _info, * FROM table ...";
This would show me the file issuing the query when I looked at mytop, and it didn't foil the MySQL query cache like using
/* __FILE__.'.'.__LINE__ */
at the start of my query would.
I suppose another couple of things I can do, with regard to the general memory issue, as opposed specifically to MySQL, and particularly within the context of our own custom code, would be to wrap our code with calls to one or the other of the following PHP built-in functions:
memory_get_usage
memory_get_peak_usage
In particular since I am currently working on logging from some custom code, I can log the memory usage while I'm at it