I'm maintaining an inherited site built on Drupal. We are currently experiencing "too many connections" to the database.
In the /includes/database.mysql.inc file, #mysql_connect($url['host'], $url['user'], $url['pass'], TRUE, 2) (mysql_connect() documentation) is used to connect to the database.
Should $new_link = TRUE be used? My understanding is that it will "always open a new link." Could this be causing the "too many connections"?
Editing core is a no no. You'll forget you did, upgrade the version, and bam, changes are gone.
Drupal runs several high-performance sites without problems. For instance, the Grammy Awards site switched to Drupal this year and for the first time the site didn't go down during the cerimony! some configuration needs tweaking on your setup. Probably mysql.
Edit your my.cfg and restart your mysql server (/etc/my.cfg in fedora, RH, centos and /etc/mysql/my.cfg on *buntu)
[mysqld]
max_connections=some-number-here
alternatively, to first try the change without restarting the server, login to mysql and try:
show variables like 'max_connections' #this tells you the current number
set global max_connections=some-number-here
Oh, and like another person said: DO. NOT. EDIT. DRUPAL. CORE. It does pay off if you want to keep your site updated, may cause inflexible headache and bring you a world of hurt.
MySQL, just like any RDBMS out there will limit the amount of connections that it accepts at any time. The my.cnf configuration file specifies this value for the server under the max_connections configuration. You can change this configuration, but there are real limitations depending on the capacity of your server.
Persistent connections may help reducing the amount of time it takes to connect to the database, but it has no impact on the total amount of connections MySQL will accept.
Connect to MySQL and use 'SHOW PROCESSLIST'. It will show you the currently open connections and what they do. You might have multiple connections sitting idle or running queries that take way too long. For idle connections, it might just be a matter of making sure your code does not keep connections open when they don't need them. For the second one, they may be parts of your code that need to be optimized so that the queries don't take too long.
If all connections are legitimate, you simply have more load than your current configuration allows for. If you MySQL load is low even with the current connection count, you can increase it a little and see how it evolves.
If you are not on a dedicated server, you might not be able to do much about this. It may just be someone else's code causing trouble.
Sometimes, those failures are just temporary. When it fails, you can simply retry the connection a few milliseconds later. If it still fails, it might be a problem and stopping the script is the right thing to do. Don't put that in an infinite loop (seen it before, terrible idea).
FYI: using permanent connections with Drupal is asking for trouble. Noone uses that as far as I know.
The new_link parameter only has effect, if you have multiple calls to mysql_connect(), during 1 request, which is probably not the case here.
I suspect it is caused by too many users visiting your site, simultaneously, because for each visitor, a new connection to the DB will be made.
If you can confirm that this is the case, mysql_pconnect() might help, because your problem is not with the stress on your database server, but the number of connections. You should also read Persistent database connections to see if it is applicable to your webserver setup, if you choose to go this route.
Related
We're using PHP on AWS, with RDS/Aurora.
This works by exposing an endpoint for the cluster, which was CNAME records to the currently active mysql nodes.
As we add/remove reader nodes, in the case of a failover, this endpoint is updated automatically, with a 5 second TTL.
As such, our app should see and respond to the new nodes very quickly.
We're noticing after a failover, we get 'Mysql has gone away' for much longer than the 5 second period. We've had instances of this being 30 minutes, at which point we restarted Apache and it resolved the issue.
It seems as though somewhere in the application, the database is not querying the endpoint DNS and resolving the new endpoints, therefore still pointing at a node which is no longer there.
We do use Persistent connections (for performance), which was the obvious culprit, however we've then tested with these turned off, and same behaviour exists.
We're using PHP 7.1, with Mysqli. We have a singleton class around the mysqli connection, but even if this kept the same connection open it would only last the time a single script executed, which is typically a few milliseconds.
Any guidance as to where the caching may be occurring?
It's unclear if your issue is DNS related on the remote services or caching on your own (AWS) local network/services. This is the first thing to look into.
To my knowledge Linux does not cache DNS lookups and nor does Apache/PHP (unless you're using mod_proxy, in which case look into disablereuse Off setting). With this in mind I expect that your Apache service restart causing it start working was likely a coincidence.
My first suggestion would be to force a fail-over and then immediately check with the name servers from several different geographical locations and using a terminal from your AWS server to see how long it takes the same servers to report updated results. The chances are name servers are simply ignoring your TTL, or perhaps just 'treating it as a suggestion'.
The long and the short of it is that DNS TTL is just a suggestion to resolving name servers as to how long to cache. There is nothing to enforce name servers to actually abide by what you set. And the reality of it is, many name services don't follow your setting exactly or even at all.
If the name-servers are updating as quickly as expected elsewhere, but not on your AWS server and mysql still can't connect; this suggests caching somewhere on your server or more likely within the AWS network. Unless the caching is directly on your server, which as discussed above I believe is unlikely, I doubt there is much you can do about this.
Ultimately updating a DNS record and using a low TTL as a fail-over solution is likely never going to achieve consistent <1 min fail-over speeds.
You may want to look into alternative methods such as ClusterControl or a proxy method such as ProxySQL.
On my website, when an user opens his profile or any other page, (almost all pages use data from mysql), my website makes around 50 connections to mysql in loading the page. This is the case on my development server, running elementary OS.
When I came to know about persistent connections in mysql, I was puzzled. If I am to run this website on a VPS (with low RAM in starting), and considering the overhead produced by a large number of mysql connections, will using persistent connections improve my website's performance?
Currently, I start and end a connection in every function. Is there a better way to connect to mysql?
And, taking into account that if 100 users are using my website simultaneously, what will be the performance if each page makes around 50-60 connections?
You asked so I will answer. You are doing this incorrectly. You should start the processing on each page request (each externally-accessible .php file) by using a common function to establish a single database connection, then you should reuse that connection.
You are getting away with this because you're probably using an automatic connection pool built in to your php database-access library, and because you have not yet scaled up your application.
You won't be able to scale this up very far using this multiconnection strategy because it will perform very badly indeed when you add users.
There are lots of examples of working open-source php-based web app systems you can look at. WordPress is an example. You'll find that most of them start by opening a database connection and storing its handle in a global variable.
You asked:
According to CBroe's comment, I changed my strategy. Actually, I am
using multiple database connections, but the functions are same (don't
ask why lol). So if I open connections on start, and then pass the
handler to the function, will that be an improvement?
Yes, that will be fine. You need to avoid churning connections to get best performance.
If you need connections to more than one different database, you can open them all. But it sounds like you need to hit just one database.
PHP doesn't have significant overhead when passing a handler to a function, so don't worry about that.
As explained wonderfully by Ollie Jones, I opened the connection on start, and my connections dropped from 50-60 per page to 1 per page. Although I don't see any change in performance on my local development server, this will surely we a great improvement when its on a live server. There is no need for me to use persistent connections yet.
I have a webapp written in PHP that currently creates a DB connection (using mysqli_connect) on every page to pull data from the database.
Recently, my website has slowed down (with some traffic increase), and I was wondering if the creation of so many connections - one for every user that is on any page - is causing the slow down?
Is there any fix for this?
Is it possible to create one sharable connection for the server? I know this is possible in python, but I do not know how I would implement such a model in PHP.
Note: My site is on BlueHost...I don't know if that also makes a difference.
Well two things to do.
Setup the slow query log in MySQL and see if there are queries that
are slow. Optimize these slow queries by using the EXPLAIN command
to identify missing indexes etc.
Setup connection pooling to eliminate opening a connection all the
time. See this link Connection Pooling In PHP for more information.
You can prepend host by "p:" when passing it to mysqli_connect to open a persistent connection.
You can read about Persistent Connections from the links below:
http://us2.php.net/manual/en/features.persistent-connections.php
http://php.net/manual/en/mysqli.persistconns.php
http://us2.php.net/manual/en/function.mysql-pconnect.php
But I would strongly suggest NOT to use Persistent Connections.
You will need to find out what is actually slowing your website.
Ping a website that loads fast for you. eg. google.com.
Ping your webserver, if the ping difference is a lot then you should ask BlueHost to resolve it.
Check whether there is any lag between your Web Server and Database Server. They could probably be:
on the same machine (mostly no lag)
on different machines in the same local network (mostly no lag unless there is a LAN problem, ask BlueHost)
on different machines in different networks (there could be issues, ask BlueHost if they could shift the Database Server within the same Local Network)
If everything above is fine and the pages are still loading slowly.
You could try explicitly calling mysqli_close() after your DB work in the page is done. This will free up your connection immeditely instead of waiting for the page to fully execute. You will then need to figure what is slowing the page after your DB work is over.
You can use mictotime() on your slow pages to see what code is slowing it down.
You could use this link: Accurate way to measure execution times of php scripts
I have the following question regarding the memcached module in PHP:
Intro:
We're using the module to prevent the same queries from being sent to the Database server, on a site with 500+ users in every moment.
Sometimes (very rarely) the memcahed process defuncts and all active users start generating queries to the database, so everything stops working.
Question:
I know, that memcached supports multiple servers, but I want to know what happens when one of them dies? Is there some sort of balancer background or something, that can tell Ow! server 1 is dead. I'll send everything to server 2 until the server 1 goes back online. or the load is being sent equally to each one?
Possible sollutions:
I need to know this, because if it's not supported our sysadmin can set the current memcached server to be a load ballancer and to balance the load between several other servers.
Should I ask him to create the load-balancing manualy or is this feature supported by default and what are the risks for both methods?
Thank you!
You add multiple servers in your PHP script, not in Memcache's configuration.
When you use Memcached::addServers(), you can specify a weight for every server. In your case, you might set one Memcache server to be higher than the other and only have the second act as a failover.
Using Memcached::setOption(), you can set how often a connection should be retried and set the timeout. If you know your Memcache servers die a lot, it might be worth it to set this lower than the defaults, but it shouldn't be necessary.
I'm implementing an application that will have a lot of clients querying lots of small data packages from my webserver. Now I'm unsure whether to use persistent data connections to the database or not. The database is currently on the same system as the webserver and could connect via the socket, but this may change in the near future.
As I know a few releases of PHP ago mysqli_pconnect was removed because it behaved suboptimally. In the meantime it seems to be back again.
Based on my scenario I suppose I won't have an other chance to handle thousands of queries per minute but with loads of persistent connections and a MySQL configuration that reserves only little resources per connection, right?
Thanks for your input!
What happened when you tested it?
With the nest will in the world, there's no practical way you can convey all the informaton required for people to provide a definitive answer in a SO response. However usually there is very little overhead in establishing a mysql connection, particularly if it resides on the same system as the database client (in this case the webserver). There's even less overhead if you use a filesystem rather than a network socket.
So I'd suggest abstracting all your database calls so you can easily switch between connection types, but write your system to use on-demand connections, and ensure you code explicitly releases the connection as soon as practical - then see how it behaves.
C.
Are PHP persistant connections evil?
The problem is there can be only so
many connections active between Host
“Apache” and Host “MySQL”
Persistant connections usually give problems in that you hit the maximum number of connections. Also, in your case it does not give a great benefit since your database server is on the same host. Keep it to normal connections for now.
As they say, your mileage may vary, but I've never had good experiences using persistent connections from PHP, including MySQL and Oracle (both ODBC and OCI8). Every time I've tested it, the system fails to reuse connections. With high load, I end up hitting the top limit while I have hundreds of idle connections.
So my advice is that you actually try it and find out whether your set-up is reusing connections properly. If it isn't working as expected, it won't be a big lose anyway: opening a MySQL connection is not particularly costly compared to other DBMS.
Also, don't forget to reset all relevant settings when appropriate (whatever session value you change, it'll be waiting for you next time to stablish a connection and happen to reuse that one).