(Not a Linux guru!) At work we have for the past month had a serious issue that we haven't been able to solve. From a server running Ubuntu-Linux we are using PHP to connect to a server running Microsoft SQL Server 2005. This has been working without problems for a long time. At the start of February 2013 we moved the SQL Server machine into the same core network as the Linux-machine, which involved changing IP-address on the Windows-server running SQL Server 2005.
After this operation we've had issues connecting to the SQL Server from this particular Linux-machine. Maybe one out of 1000 connections fails with the following error message:
PHP Warning: mssql_connect(): Unable to connect to server
I'm not aware of any other changes to either of these two servers. We've tried to look at 'everything'.
FreeTDS
php.ini adjustments
IP-address in SQL Server Configuration Manager
Unlimited number of concurrent connections in SQL Server
Network and firewall issues (No packet loss during 24 hours)
Any suggestions to how we can pursue error searching for this problem? Any more info anyone would like to know about this setup? It's kinda serious for us since many of our cronjobs fail to execute normally due to this random connection problem.
Rgds,
Sven David, Norway
Very difficult to diagnose from an external perspective, however, it feels like a networking issue rather than php/sql config. Presumably, because it's intermittent, it's not mssql functions/extension. Very little is truly random!
Perhaps approach this by writing a script that monitors the servers (both hosts) vital stats - like a heartbeat: IPs, DNS, DHCP?, requests etc and setting it to record anomalies/alert you when any element changes on either host or connect requests fail...
Probably worth posting on other SE sites too - not just Stackoverflow.
Related
Good morning to everybody, I would like to ask about a problem I have and which I'm not able to solve it.
I have two servers, one is the web server (it contains a large web application) and the other one is the BD server.
The problem is that both are virtualized in a physical server with VMWare and they were running correctly until two weeks ago. In the start of this month we noticed that the web application ran very slow and we started to investigate what was the problem. We have tried a lot of things and we do not know what is the problem and, of course, how to solve it.
Both servers has an internal IP and only web server is accessible from Internet. Only web server access to BD server in order to get the results of queries. It's true that web server and BD server has an older versions of PHP and MySQL respectively.
We did the following tests:
Analyze the consumption of both servers. They are in 1% of memory, swap and al types of consumptions. Our physicial server is new from a year ago and both virtual servers does not use more than 5% of their resources.
Reboot both servers
Reboot physical server (VMWare server)
Restore a backup from 1 and 2 months ago of both servers to discard code and BD data errors
Review code (we do not touch code from before the error)
We did some queries from terminal's webserver and they were fast. We looked for the BD log and we could see that queries have been doing one by one in a "slow" velocity (0,5s for query aprox, depending on the query it can be more than a second).
We suspect that PHP is doing something bad, but we do not touch the code and we do not update PHP version or MySQL version. We want to try to update MySQL version in a new virtual machine and migrate all the data there, but we think it will not solve the problem.
The connection between servers is perfect and we think that layer 2 should not be the problem. In the same webserver we have another web application (Moodle) that connects to this BD server too, and it does not have this problem...
What can be the problem? It's very strange this change of behaviour of the web application. We were on holidays on August and we returned and we found this problem.
For more information we use PHP 5.6.40 (webserver) and MySQL Ver 14.14 Distrib 5.1.60, readline 5.1 (BD server). (yes, we know they are old versions but the web application and BD are old too)
I hope someone can help us, we are a little bit lost.
Thanks for your help!
I'd try updating your servers for one, issues like these are commonly caused by older software.
I'd also start logging or looking into loads so you can determine whatever causes the slow speeds you're getting.
I'm using mysqli connection over ssl,
so using in code following:
$mdbconnection = mysqli_init();
$mdbconnection->ssl_set('/ssl/xyz.pem', '/ssl/abc-cert.pem', NULL, NULL, 'AES256-SHA');
$mdbconnection->real_connect(SV_SERVER, SV_MYSQL_LOGIN, SV_MYSQL_PASSWORD, SV_MYSQL_DATABASE);
I noted down that I am not able to connect over mysql server all the time.
if trying 100 times failing 3 time.
and getting following error
echo($mdbconnection->connect_errno);
printing:
2003
and if I am trying second time after failing it{echo($mdbconnection->connect_errno);} will give always 1045.
I am trying to solve this bug.
Any ideas would be appreciated.
The 2003 error code means the connection couldn't be established:
https://dev.mysql.com/doc/refman/5.5/en/error-messages-client.html
Error code 1045 means the client connected but the server rejected the credentials (access denied):
https://dev.mysql.com/doc/refman/5.5/en/error-messages-server.html
I'd say that if you're getting the 2003 error code a few times out of every 100 tries, it sounds like some of the possibilities could be:
Some intermediate application or piece of hardware might be seeing your attempts as a possible flood attack (DDoS) and could be intercepting and blocking the connection (if you're connecting 100 times really quickly)
You might have some network stability issues on the client side. It might be worth the time to run Wireshark and reproduce the problem with it running in order to capture the problem and see if it shows anything significant (e.g. DNS failure).
The MySQL server might be crashing and then restarting. Check your server-side MySQL error log (https://dev.mysql.com/doc/refman/5.7/en/error-log.html). Actually, you should really do this as a first step any time you have bizarre behavior in MySQL.
If the MySQL server log doesn't show anything, check your system logs, especially if you're logging anything like blocked firewall (iptables or whatever f/w you're using) packets and search them for your client IP around the time of the problem.
Finally, if you can reproduce this easily/quickly (e.g. make 100 connections via a script and see it fail), try eliminating SSL out of the equation and connecting without SSL and see if it gives you the same problem. If it works flawlessly without SSL, perhaps there's a problem with the SSL library that was used during the compilation of your MySQL package.
You might also want to check to see if you have any load-balancing in place that might make your script connect to different instances of the MySQL server (this is less likely, but could explain the symptoms you're seeing if one server is down and the next doesn't have the credentials you're providing)
UPDATE
So I completely disabled my server's firewall and it appears to be the culprit. I had tried to disable certain rules before but disabling the whole thing worked. Very frustrating, but problem solved I guess... I think the key indicator that it was the firewall was because it happened at exactly midnight when my server likes to apply updates and such.
This is pretty strange to me, I have a server downstairs that hosts my websites and MySQL servers and it has been running for years without many issues. I have 2 routers bridged together behind my modem and my server is behind one of them. All other devices connect via WiFi. All of the proper ports are open on the router and I have users configured in MySQL that haven't changed and have been working fine this whole time.
So last night I was working on a project and I decided to sync everything with a backup on my SkyDrive. I have a scheduled backup for MySQL that runs at midnight (daily) and it just turned over to midnight so I decided to open my network and watch the file get populated before I sent a copy to my SkyDrive. After the backup was complete (which it did successfully), I was going to continue to work on my program but all of a sudden I can no longer connect via my local network to the MySQL server. I'm using PHP and my connection string never changed and all other MySQL admin tools don't connect. The live site works fine, so MySQL was definitely running and working but no remote connections were being accepted. Why is this happening all of a sudden?
Things I've tried :
I did notice that my logs were packed full of BINLOG errors so I turned off the binlog since I recently turned it on (a couple weeks ago).
Restarting MySQL
Turning off Windows Server 2008 firewall (temporarily)
Connecting from a different device (mobile phone, tablet), no luck
Temporarily allowing port 3306 on my router
Checked server logs for intrusion attempts, none present...
Setup :
PHP 5.4 on local machine and server
Windows Server 2008 Enterprise (on server...)
MySQL Version 5.5.25a
Does anyone have any clue as to what's going on here? I'm going to reboot my server when the load is low and see if this helps any, I will update this once it comes back online.
So I completely disabled my server's firewall and it appears to be the culprit. I had tried to disable certain rules before but disabling the whole thing worked. Very frustrating, but problem solved I guess... I think the key indicator that it was the firewall was because it happened at exactly midnight when my server likes to apply updates and such.
If MySQL is dropping connections in a PHP application, and MySQL connection limit is set above the number of concurrent users in the application, which other factors can contribute to this behavior? Also, analyzing moodle logs (the only app running at these servers), yesterday I had 4 times more activity and it didn't drop anything, but today there were times where it was frustrating the number of dropped connections.
My main question here is why the database is rejecting connections when it didn't before with 4 times more activity (and everything it's the same, I didn't change anything in between).
Some background: I've got 2 servers contracted at my hosting:
shared server running Debian Linux/PHP 5.3 (FastCGI)
VPS running Debian Linux/MySQL 5.1.39
On this environment I'm running only moodle 1.9.12 (for the database connection using adoDB and persistent connections), php part on the shared server, database on the VPS. I suspect that, by PHP running on a shared server, other hosting accounts are affecting me (the database rejecting connections I mean, I really don't care about RAM/CPU).
Reading about the issue I've seen in places that persistent connections don't work well with PHP as CGI/FastCGI and that if both servers are in the local lan it really doesn't matter using persistent or not persistent connections because of the connection is going to be quick anyway. So now I'm using not persistent connections. I guess that may be part of the problem, but I fail to understand why it worked with more load. Which PHP/Apache settings are involved here?
Since your database and web server are on two different machines, there are two other possible causes: the network in between, and the network layer of the operating system.
If you did not change anything in your configuration and it worked with higher loads before, it is more likely to be an issue with the network connectivity between the two machines. Since the database machine is a VPS you also don't know how much real network load it is handling. If your ISP has competent support personnel (which unfortunately isn't always the case) it can't hurt to ask them if they have an explanation.
The same goes for your "shared" web-server. While it is unlikely, it is not impossible that it is a an issue of too many connections on that machine.
It would also help to know how exactly you are measuring dropped connections. If you are looking at the aborted-connections counter of mySQL, it is not necessarily a measure of an actual problem: http://dev.mysql.com/doc/refman/5.1/en/communication-errors.html. A user aborting a page load already may increase this counter.
If PHP throws an error, because it could not connect to the server or lost connection to the server during a query, then it is an issue.
I have a machine that is only able to connect to a 2008 SQL server sporadically. I created a monitoring script that I have automated to attempt a connection (via sqlsrv driver and php) every 15 mins. This script has ran fine for the last two weeks but recently it has been reporting a SQL server going down. On extensive investigation I am able to connect to it from every other machine consistently except for the machine automating the monitoring script. This script also connects to two other SQL servers (both are sql server 2005) without any problems. Here is what I have tried to resolve the failures:
1. Reset both machines (serveral times)
2. Attempted connections via several different logins countless times (including sa).
3. Deleted and recreated logins and attempted to connect.
4. Turned off windows firewall (SQL server is running on win2k8)
This is the error I get when I cant connect: http://pastebin.com/9d80mAyV
Typically it wont work for about 5 attempts (spaced 15 mins apart) then it will work for 1 attempt. Then it fails for another 5 attempts. There is no absolute consistency.
My thoughts is that it has blacklisted my client from connecting too often. Is this possible? How could I resolve this problem? Any other thoughts or suggestions?
A few more details needed:
What are you using to connect? is it an application or management studio (what version?)
Where are the servers located? i.e. are you on the same network?
You can try to increase your connection timeout. If you are connecting from management studio, then you can do this in the server registration properties.
Alternatively, you can setup sql monitoring on the server itself, and it will email you when it goes down.