I have a site that is about 6 years old, and in the last 6 months, we have been getting the same issue over and over with maxing out the mysql connections.
Now every couple of days the site becomes unavailable because there are no sql connections left. As it's on a shared server, I need to get the host company to flush the connections.
Here's what I tried:
Made sure every page now has a close on it
Upgraded to PHP7 and converted to mysqli
Pulled my hair out
Everywhere I look, it says not to use Persistant connections - I am not explicitly using them - and that PHP will automatically release connections after the page completes - hmmm...
The host company is usually really helpful, but not over this. So I have 2 questions...
What is causing the connections to not close either on my mysqli_close() or on PHP exit? Perhaps pages are aborting or too slow and failing?
Why don't the connections close themselves... It can be hours later that I report it to the host company and they flush the connections... Why are they not bring tidied up? In the mean time the site is unavailable...
Please help..
Cris.
Ah... I have just read the host company responses more clearly... (it's been a hectic few days) They say "The issue that is the database server on the shared hosting platform is designed to lock the database once it hits it's maximum allowed connections, so it never closes any connections once it hits 25 simultaneous connections. "
So the question is why am I hitting 25 connections? Is is that I'm holding connections open for slow loading pages and so limiting the number available?
whenever you open a mysqli connection
$mysqli = connect_db();
you have to explicitly close it or they linger around and mysql server eventually is unhappy
$mysqli->close();
While explicitly closing open connections and freeing result sets is optional, doing so is recommended. This will immediately return resources to PHP and MySQL, which can improve performance. http://php.net/manual/en/mysqli.close.php
Related
I am using PDO to connect to a Postgresql DB, using PDO::ATTR_PERSISTENT attribute which does speed up the application noticeably.
However I encountered a problem, after debugging it, found that the connections to DB die about 2 hours 11 minutes after the last SQL query executed!
The fault is at the Firewall between the servers, but the networking guy refuses to have the limit higher than 24 hours (saying it would slow down the firewall) while I need it to be disabled.
I cant refresh the pooled connections by simple usage, because I cant control or ensure that ALL the pooled connections are served on a rotating base, ensuring no connection would die (causing hard to debug issues later when there is load on the server, and some connections are dead, while others are alive).
So, does anyone have any suggestion on the problem? how to keep the pooled connections alive, other than forcing the Networking guy to disable this silly timeout rule for those 2 specific servers?
Thanks!
Use
SELECT 1
as the first query, if that fails, reconnect.
I'm having a problem that I hope someone can help me out with.
Currently, every now and again we receive an error when our scripts (Java and PHP) try to connect to the localhost mysql database.
Host 'myhost' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'.
This issue appears to mainly occur in the early hours of the morning. After alot of searching to figure out why this may be occuring I have finally come to the conclusion that it may be due to the fact our hosting company runs their backup processes around this time. My theory is that during this backup process (this is also our busiest period) we end up using up all our connections and so this error occurs.
I have talked to our hosts about changing the times these backups occur but they have stated that this is not possible and that is simply the times the backups start to ensure they are finished in time for the day (Even though we have informed them our critical period is at the precise times the backups occur).
The things I have connecting to the server are:
PHP website
PHP files run using chron jobs
A couple of java applications to run as socket listeners that listen for incoming port connections and uses the mysql database for checking user credentials and checking outstanding messages.
We typically have anywhere from 300 - 600 socket connections open at any one time and the average activity on these are about 1-3 request per second.
I have also installed monit and munin with some mysql plugins on the server in the hope they may help auto resolve this issue however these do not see to resolve the issue.
My questions are:
Is there something I can do to auto poll the mysql database so if this occurs I can auto flush the database to clear
Is this potentially even related to the server backup. It seems a coincidence it happens 95% of the time during the period the backups occur.
Any other ideas that may help. Links to other websites, or questions I could put to our host to help out.
We are currently running on a PHP Version 5.2.6-1+lenny9 server with Apache.
If any more information is required to help, please let me know. Thanks.
UPDATE:
I am operating on a shared virtual host and am pretty sure I close my website connections as I have this code in my database class
function __destruct() {
#mysql_close($this->link);
}
I'm pretty sure I'm not using persistant connections via my PHP script as I connect to the db the #mysql_connect command.
UPDATE:
So I changed the max_connections limit from 100 - 200 and I changed the mysql.persistant variable from On to Off in php.ini. Now for two nights running the server has gone done and mainly the connection to the mySql database. I have one 1GB of RAM on the server but it never seems to get close to that. Also looking at my munin logs the connections never seem to hit the 200 mark and yet I get errors in my log files something like
SQLException: Too many connections
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
SQLException: null, message from server: "Can't create a new thread (errno 12); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug.
SQLState :: SQLException: HY000, VendorError :: SQLException: 1135
We've had a similar problem with out large ecommerce installation using MySQL as a backend. I'd suggest you alter the "max_connections" setting of the MySQL instance, then (if necessary) alter the number of file descriptors using "ulimit" before starting MySQL (we use "ulimit -n 32768" in /etc/init.d/mysql).
It's been suggestion I post an answer to this question although I never really got it sorted.
In the end I ended up implementing a Java connection pooling class which enabled me to share connections whilst maintaining a upper limit on the number of max connections I wanted. It was also suggested I increase the RAM and increase the number of max connections. I did both these things although they were just bandaids to the problem. We also ended up moving hosting providers as the ones we were with were not very co-ooperative.
After these minor implementations I haven't noticed this issue occur for at least 8 months which is good enough for me.
Other suggestions over time have to also implement a Thread pooling facility, however current demand does not require this need.
I have a mongodb server in production serving on an EC2 instance. According to the mongodb official documentation, persistent DB connections should ALWAYS be used in production. I've been experimenting with about 50 persistent connections and was getting frequent connection errors (approx 33% of the time) while testing. I'm using this code:
$pid = 'db_'.rand(1,50);
$mongo = new Mongo("mongodb://{$user}:{$pass}#{$host}", array('persist' => $pid) );
Some background on the application, it's a link tracking application that is still ramping up - and is in the range of 500 - 1k writes per hour, nothing too crazy... yet.
I'm wondering if I simply need to allow more persistent connections? How does one determine the right balance of persistent connections versus server resources available?
Thanks in advance everyone.
The persist value is no longer supported as of the most recent driver (1.2.0).
Truth is, it was never really clear what it did in typical Apache+PHP setups. There are several comments on the Google Groups and elsewhere asking for detail, but I did not any evidence that persist or persistent was ever tested with any depth.
Instead, it's all been replaced by connection pooling "out of the box". The connection pooling has obviously been through some changes within the 1.2 line with the addition of the MongoPool class.
There is still no detailed explanation of how the pooling works with Apache, but at least you don't have to worry about persist.
Now despite all of this mess, I have handled 1000 times that traffic on a single MongoDB server via the PHP driver without lots of connection problems.
Are you catching the exceptions?
Can you provide more details about the exact exception?
There may be a code solution.
Are you opening a new connection for each PHP page request, or using a connection pool with 50 persistent connections? If you're opening a new connection each time then you might be quickly running out of resources.
Each connection uses an additional thread on the server, so you could be hitting a limit on the number of threads of network connections, check your server logs in /var/lib/mongodb for errors.
If you're using the official MongoDB PHP driver, then as far as I know it should handle connection pooling for you automatically. If you're connecting to Mongo from 50 separate clients, then consider putting a queue in front of Mongo to buffer the writes.
http://php.net/manual/en/mongo.connecting.php
without Persistent Connections x1000
It takes approximately 18 seconds to execute
Persistent
...it takes less than .02 seconds
Trying to separate out my LAMP application into two servers, one for php and one for mysql. So far the application connects locally through a file socket and works fine.
I'm worried about the number connections I can establish if it is over the network. I have been testing tcp connections on unix for benchmark purposes and I know that you cannot exceed a certain amount of connections per second otherwise it halts due to the lack of resources (be it sockets, or file handles or whatever). I also understand that php does not implement connection pooling so for each page load a new connection over the network must be made. I also looked into pconnect for php and it seems to bring more problems.
I know this is a very very common setup (php+mysql), can anyone provide some typical usage and statistics they get out of their servers? Thanks!
The problem is not related to running out of connections allowed my MySQL. The main problem is that unix cannot very quickly create and tear down tcp connections. Sockets end up in TIME_WAIT and you have to wait for a period before you free up more sockets to connect again. These two screenshots clearly shows this pattern. MySQL does work up to a certain point and then pauses because the web server ran out of sockets. After certain amount of time passed, the web server was able to make new connections.
alt text http://img35.imageshack.us/img35/3809/picture4k.png
alt text http://img35.imageshack.us/img35/4580/picture2uyw.png
I think the limit is at 65535. So you'd have to have 65535 connections at the same time to hit that limit since a regular mysql connection closes automatically.
mysql_connect()
Note: The link to the server will be closed as soon as the execution of the script ends, unless it's closed earlier by explicitly calling mysql_close().
But if you're using a persistent mysql connection, then you can run into trouble.
Using persistent connections can require a bit of tuning of your Apache and MySQL configurations to ensure that you do not exceed the number of connections allowed by MySQL.
Each MySQL connection actually uses several meg of ram for various buffers, and takes a while to set up, which is why MySQL is limited to 100 concurrent open connections by default. You can up that limit, but it's better to spend your time trying to limit concurrent connections, via various methods.
Beware of raising the connection limit too high, as you can run out of memory (which, I believe, crashes mysql), or you may push important things out of memory. e.g. MySQL's performance is highly dependent on the OS automatically caching the data it reads from disk in memory; if you set your connection limit too high, you'll be contending for memory with the cache.
If you don't up your connection limit, you'll run out of connections long before your run out of sockets/file handles/etc. If you do increase your connection limit, you'll run out of RAM long before you run out of sockets/file handles/etc.
Regarding limiting concurrent connections:
Use a connection pooling solution. You're right, there isn't one built in to PHP, but there are plenty of standalone ones out there to choose from. This saves expensive connection setup/tear down time.
Only open database connections when you absolutely need them. In my current project, we automatically open a database connection when the first query is issued, and not a moment before; we also release the connection after we've done all our database work, but before the page's HTML is actually generated. The shorter the period of time you hold connections open, the fewer connections will be open simultaneously.
Cache what you can in a lighter-weight solution like memcached. My current project temporarily caches pages displayed to anonymous users (since every anonymous user gets the same HTML, in the end -- why bother running the same database queries all over again a few scant milliseconds later?), meaning no database connection is necessary at all. This is especially useful for bursts of anonymous traffic, like a front-page digg.
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 7 years ago.
Improve this question
I have this doubt, I've searched the web and the answers seem to be diversified. Is it better to use mysql_pconnect over mysql_connect when connecting to a database via PHP? I read that pconnect scales much better, but on the other hand, being a persistent connection... having 10 000 connections at the same time, all persistent, doesn't seem scalable to me.
Thanks in advance.
Persistent connections should be unnecessary for MySQL. In other databases (such as Oracle), making a connection is expensive and time-consuming, so if you can re-use a connection it's a big win. But those brands of database offer connection pooling, which solves the problem in a better way.
Making a connection to a MySQL database is quick compared to those other brands, so using persistent connections gives proportionally less benefit for MySQL than it would for another brand of database.
Persistent connections have a downside too. The database server allocates resources to each connection, whether the connections are needed or not. So you see a lot of wasted resources for no purpose if the connections are idle. I don't know if you'll reach 10,000 idle connections, but even a couple of hundred is costly.
Connections have state, and it would be inappropriate for a PHP request to "inherit" information from a session previously used by another PHP request. For example, temporary tables and user variables are normally cleaned up as a connection closes, but not if you use persistent connections. Likewise session-based settings like character set and collation. Also, LAST_INSERT_ID() would report the id last generated during the session -- even if that was during a prior PHP request.
For MySQL at least, the downside of persistent connections probably outweighs their benefits. And there are other, better techniques to achieve high scalability.
Update March 2014:
MySQL connection speed was always low compared to other brands of RDBMS, but it's getting even better.
See http://mysqlserverteam.com/improving-connectdisconnect-performance/
In MySQL 5.6 we started working on optimizing the code handling connects and disconnects. And this work has accelerated in MySQL 5.7. In this blog post I will first show the results we have achieved and then describe what we have done to get them.
Read the blog for more details and speed comparisons.
Basically you have to balance the cost of creating connections versus keeping connections. Even though MySQL is very fast at setting up a new connection, it still costs -- in thread setup time, and in TCP/IP setup time from your web server. This is noticeable on a high-enough traffic site. Unfortunately, PHP does not have any controls on the persistence of connections. So the answer is to lower the idle timeout in MySQL a long way (like down to 20 seconds), and to up the thread cache size. Together, this generally works remarkably well.
On the flip side, your application needs to respect the state of the connection. It is best if it makes no assumptions about what state the session is in. If you use temporary tables, then using CREATE IF NOT EXISTS and TRUNCATE TABLE helps a lot, as does naming them uniquely (such as including as userid). Transactions are bit more problematic; but your code can always do ROLLBACK at the top, just in case.
mysql_connect() and mysql_pconnect() both are working for database connection but with little difference. In mysql_pconnect(), p stands for persistance connection.
When we are using mysql_connect() function, every time it is opening and closing the database connection, depending on the request.
But in case of mysql_pconnect() function:
First, when connecting, the function would try to find a (persistent) connection that's already open with the same host, username and password. If one is found, an identifier for it will be returned instead of opening a new connection.
Second, the connection to the SQL server will not be closed when the execution of the script ends. Instead, the connection will remain open for future use (mysql_close() will not close connection established by mysql_pconnect()).
mysql_pconncet() is useful when you have a lot of traffice on your site. At that time for every request it will not open a connection but will take it from the pool. This will increase the efficiency of your site. But for general use mysql_connect() is best.
It's very unlikely that you'll reach 10000 connections. Anyhow, go to the official source. (Emphasis mine).
If persistent connections don't have
any added functionality, what are
they good for?
The answer here is extremely simple --
efficiency. Persistent connections are
good if the overhead to create a link
to your SQL server is high. Whether or
not this overhead is really high
depends on many factors. Like, what
kind of database it is, whether or
not it sits on the same computer on
which your web server sits, how
loaded the machine the SQL server sits
on is and so forth. The bottom line
is that if that connection overhead is
high, persistent connections help you
considerably. They cause the child
process to simply connect only once
for its entire lifespan, instead of
every time it processes a page that
requires connecting to the SQL server.
This means that for every child that
opened a persistent connection will
have its own open persistent
connection to the server. For example,
if you had 20 different child
processes that ran a script that made
a persistent connection to your SQL
server, you'd have 20 different
connections to the SQL server, one
from each child.
Note, however, that this can have some
drawbacks if you are using a database
with connection limits that are
exceeded by persistent child
connections. If your database has a
limit of 16 simultaneous connections,
and in the course of a busy server
session, 17 child threads attempt to
connect, one will not be able to. If
there are bugs in your scripts which
do not allow the connections to shut
down (such as infinite loops), the
database with only 16 connections may
be rapidly swamped. Check your
database documentation for information
on handling abandoned or idle
connections.
MYSQL_CONNECT()
1.mysql_connect can be used to close the connection.Every time it is opening and closing the database connection, depending on the request .
2.Here database is opened everytime when the page is loaded in MYSQL connect
3.When the page is loaded, the database is loaded everytime
4.It is used to close the connection
Example:
<?php $conn = mysql_connect(‘host’, ‘mysql_user’, ‘mysql_password’); if(!$conn){ die(‘Could not connect: ‘ . mysql_error()); } echo ‘Connected successfully’; mysql_close($conn); ?>
Description:
host: Specifies a host name or an IP address like localhost.
mysql_user: Specifies the MySQL username
mysql_password: Specifies the MySQL password
MYSQL_PCONNECT()
1.We use the mysql_pconncet(), it initially tries to find an open persistent connection.
2.The mysql_pconncet() opens persistant connection
3.The mysql_pconnect() does not support the close connection
4.mysql_pconnect() cannot close the connection. Here open a persistant connection to the database
5.Here database need not be connected everytime.
6.The database need not be connected every time in mysql_pconncet().
more details:http://prittytimes.com/difference-between-mysql_connect-and-mysql_pconnect/