Recently few days I run into a weird problem of php not able to connect to mysql on RDS, while manual connections are fine.
So I have several php scripts sitting on our server which is hosted on EC2, and these scripts are called by the clients to communicate with a mysql server sitting on RDS. The scripts are setup so that if any connection error occurs, they would send email notification to me.
So for the recent few days I'm receiving thousands of error emails from my scripts, at the first few days the errors were simple saying the server host is not recognized, but I am sure the address is correct. And now the errors are almost all saying "Host XXX is blocked because of many connection errors, unblock with mysqladmin flush-hosts..." If I flush the hosts, the errors come back to me after a while.
What confuses me the most is that, I cannot reproduce the problem! I tried connecting to the RDS server from my EC2 server through command line and through test php scripts... They all work! If I manually invoke the scripts sending me error emails, they work too! Looking at the RDS monitor, all stat looks normal, CPU is bouncing around 50%, and plenty of hard drive space.
So I'm totally confused right now... Can anyone provide me some insight about how to solve this problem? I don't even know where to look at now...
you need to check your connection to the mysql server and check for errors. Until then you can cron a "flush hosts" to temporary fix the issue. Be aware that this should be used only as a temporary fix.
Related
Recently, from few weeks ago I got err_connection_timed_out just sometimes while working with my website.
and i am sure, my users get this error too.
When i see at error log, there are a few of message like below
PHP Warning:mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock
I changed 'localhost' to '127.0.0.1' for TCP connection instead of Unix socket, but no success.
My server host admin do not accept any problem at server side (like always!).
I asked them to restart Mysql, but they do not accept because it is a share server.
It is a Directadmin server and they recommend switch to Cpanel as the last chance.
Do you have any experience and solution for this problem?
EDIT: The main problem is err_connection_timed_out which i get in browser and i don't know if it relates to Mysql connection error in log file.
Switching might have positive effects because it's differently setup.
But if it happens sometimes it is probably the MYSQL server is burping.
Lately it's normal that the services has limit per user; so you can try to enable cache (of that is possible); to see whether it's less frequent (might be some queries that is running long).
do you have any cronjobs running or other processes around the time you receive those errors?
Normally this problem is caused due to MySQL server timing-out database connections is faster than the Stash connection pool can notice. This exhausts the Stash connection pool as it will keep its connections open while the MySQL side of the connections is already closed.
Easiest solution.
You can fix it by simply change this parameter (wait_timeout) at the Mysql config.
More difficult but better
Also you can close all connections in your code after you got the values. And finally you can change your connection to your database in order to reuse an existing connections.
And last but not least
Try a vps you got out there too many oferts, even for free. You wouldn't be able to change anything in a shared host sorry. :(
I have a problem with a PHP script triggering a "connection refused" message. Im trying to execute file_get_contents('http://some_domain'). I have also tried fsockopen(), socket_create() and curl_exec() but they all result in "connection refused" on some specific sites that I want to reach. (i.e. its not a general problem).
The resources do exist and I can contact them via my browser.
If I run the PHP on my localhost server, it connects OK
The problem is only apparent if I run the code on either of my remote servers
The problem only occurs with some sites, not all sites
The code worked OK until a few weeks ago
A common factor is that the three sites Im trying to reach are all running web servers behind domestic broadband routers that I installed (but with different broadband providers). But - as I said above - I can access them all from my browser, and the code worked until a few weeks ago. As far as I can see its not a firewall issue in the routers.
Im not sure what all that means - could there be some firewall issue on some intervening machine? How could I tell?
The answer to this turned out to be nothing but a server firewall problem. A new rule set had been loaded into iptables. Hopefully this will be useful to future enquirers - check iptables and your server log and save yourself a lot of time!
I'm using mysqli connection over ssl,
so using in code following:
$mdbconnection = mysqli_init();
$mdbconnection->ssl_set('/ssl/xyz.pem', '/ssl/abc-cert.pem', NULL, NULL, 'AES256-SHA');
$mdbconnection->real_connect(SV_SERVER, SV_MYSQL_LOGIN, SV_MYSQL_PASSWORD, SV_MYSQL_DATABASE);
I noted down that I am not able to connect over mysql server all the time.
if trying 100 times failing 3 time.
and getting following error
echo($mdbconnection->connect_errno);
printing:
2003
and if I am trying second time after failing it{echo($mdbconnection->connect_errno);} will give always 1045.
I am trying to solve this bug.
Any ideas would be appreciated.
The 2003 error code means the connection couldn't be established:
https://dev.mysql.com/doc/refman/5.5/en/error-messages-client.html
Error code 1045 means the client connected but the server rejected the credentials (access denied):
https://dev.mysql.com/doc/refman/5.5/en/error-messages-server.html
I'd say that if you're getting the 2003 error code a few times out of every 100 tries, it sounds like some of the possibilities could be:
Some intermediate application or piece of hardware might be seeing your attempts as a possible flood attack (DDoS) and could be intercepting and blocking the connection (if you're connecting 100 times really quickly)
You might have some network stability issues on the client side. It might be worth the time to run Wireshark and reproduce the problem with it running in order to capture the problem and see if it shows anything significant (e.g. DNS failure).
The MySQL server might be crashing and then restarting. Check your server-side MySQL error log (https://dev.mysql.com/doc/refman/5.7/en/error-log.html). Actually, you should really do this as a first step any time you have bizarre behavior in MySQL.
If the MySQL server log doesn't show anything, check your system logs, especially if you're logging anything like blocked firewall (iptables or whatever f/w you're using) packets and search them for your client IP around the time of the problem.
Finally, if you can reproduce this easily/quickly (e.g. make 100 connections via a script and see it fail), try eliminating SSL out of the equation and connecting without SSL and see if it gives you the same problem. If it works flawlessly without SSL, perhaps there's a problem with the SSL library that was used during the compilation of your MySQL package.
You might also want to check to see if you have any load-balancing in place that might make your script connect to different instances of the MySQL server (this is less likely, but could explain the symptoms you're seeing if one server is down and the next doesn't have the credentials you're providing)
UPDATE
So I completely disabled my server's firewall and it appears to be the culprit. I had tried to disable certain rules before but disabling the whole thing worked. Very frustrating, but problem solved I guess... I think the key indicator that it was the firewall was because it happened at exactly midnight when my server likes to apply updates and such.
This is pretty strange to me, I have a server downstairs that hosts my websites and MySQL servers and it has been running for years without many issues. I have 2 routers bridged together behind my modem and my server is behind one of them. All other devices connect via WiFi. All of the proper ports are open on the router and I have users configured in MySQL that haven't changed and have been working fine this whole time.
So last night I was working on a project and I decided to sync everything with a backup on my SkyDrive. I have a scheduled backup for MySQL that runs at midnight (daily) and it just turned over to midnight so I decided to open my network and watch the file get populated before I sent a copy to my SkyDrive. After the backup was complete (which it did successfully), I was going to continue to work on my program but all of a sudden I can no longer connect via my local network to the MySQL server. I'm using PHP and my connection string never changed and all other MySQL admin tools don't connect. The live site works fine, so MySQL was definitely running and working but no remote connections were being accepted. Why is this happening all of a sudden?
Things I've tried :
I did notice that my logs were packed full of BINLOG errors so I turned off the binlog since I recently turned it on (a couple weeks ago).
Restarting MySQL
Turning off Windows Server 2008 firewall (temporarily)
Connecting from a different device (mobile phone, tablet), no luck
Temporarily allowing port 3306 on my router
Checked server logs for intrusion attempts, none present...
Setup :
PHP 5.4 on local machine and server
Windows Server 2008 Enterprise (on server...)
MySQL Version 5.5.25a
Does anyone have any clue as to what's going on here? I'm going to reboot my server when the load is low and see if this helps any, I will update this once it comes back online.
So I completely disabled my server's firewall and it appears to be the culprit. I had tried to disable certain rules before but disabling the whole thing worked. Very frustrating, but problem solved I guess... I think the key indicator that it was the firewall was because it happened at exactly midnight when my server likes to apply updates and such.
I have a server that's returning an unexpected, intermittent error and wondering if anyone has either experienced it before or can guess what might be going on. My search has so far has been unfruitful.
Im connecting to a remote mysql server using mysql_connect() in the usual way, but for the last 2 weeks (couple of times a day) the connection to the database fails and I get an unable to connect to socket error?
Error: 2002 - Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Obviously this is unexpected as Im trying to connect to a remote host. Does mysql_connect default to localhost if the remote host is unavailable? I've not known it to do that, and nothing in the docs that say it does.
There doesn't seem to be any pattern to the failed connections, various files and it happens at all sorts of time of day. Im thinking it might be network traffic related, because if you refresh the page straight away it connects fine, but am confused as to why Im getting an unable to connect to socket error.
the database provider has checked and are unable to find an issue with the server itself.
anyone have any ideas?
Im sure it doesnt default to localhost, and im also pretty sure that it is the remote server sending you the error message. So it's "Local" on your mysql server, not local on "your" server.
In this case its probably due to downtime on the remote server.
Gr.
ok, facepalm moment... after this hitting me for ages I finally found why mysql_error() was reporting the socket error. in my code there was a mysql_select_db() that I didnt notice and it was that, thats reporting the socket error as there is no open connection for it to use.
actual error the mysql_connect is reporting is (2003) Can't connect to MySQL server on 'mysql.hostname.tld'. Im still leaning towards some sort of DNS lookup/network traffic issue, Im currently using the IP address to connect to see if that errors.
But as far as this question is concerned, the answer is "Im a numpty".
Have a look at this
Had a similar thing a while back.
Are you connecting from home? The reason I ask is that my ISP likes to mess around re-assinging IP addresses to my home account, and I have an IP authorization on my remote MySQL.
I think I opened up the IP connnection limitation thing to a string that was IP Addresses for my region and that solved the problem.
If you're running Windows, there might be too many connections already, if so go to http://dev.mysql.com/doc/refman/5.5/en/can-not-connect-to-server.html#can-not-connect-to-server-on-windows and have a read.
Else, perhaps someone someone may have removed the Unix socket file that mysqld uses (/tmp/mysql.sock by default). For example, you might have a cron job that removes old files from the /tmp directory.