Webapp goes offline randomly, the offline time is less than 2minutes. (Because the downtime checker intervals are set for 2 min test and it isn't reporting this). But users are complaining about unavailability.
Getting various alerts in phpMyadmin doubts this is something related to a MySQL connection drop.
Here are the alerts that are displayed in phpmyadmin:
Configuration.
ubuntu 16.04LTS up to date
CloudFlare enabled
PHP version: 7.0.30-0ubuntu0.16.04.1
Apache
MYSQL Server version: 5.7.23-0ubuntu0.16.04.1
Looking these values it seems your tables aren't indexed (properly). Without knowing enough about your application my best bet would be:
Restart MySQL server (this is a must)
Monitor these values after restart
Enable and examine slow query logs, thus identifying queries causing these issues
Run EXPLAIN on these queries and try to introduce proper indices
Related
Good morning to everybody, I would like to ask about a problem I have and which I'm not able to solve it.
I have two servers, one is the web server (it contains a large web application) and the other one is the BD server.
The problem is that both are virtualized in a physical server with VMWare and they were running correctly until two weeks ago. In the start of this month we noticed that the web application ran very slow and we started to investigate what was the problem. We have tried a lot of things and we do not know what is the problem and, of course, how to solve it.
Both servers has an internal IP and only web server is accessible from Internet. Only web server access to BD server in order to get the results of queries. It's true that web server and BD server has an older versions of PHP and MySQL respectively.
We did the following tests:
Analyze the consumption of both servers. They are in 1% of memory, swap and al types of consumptions. Our physicial server is new from a year ago and both virtual servers does not use more than 5% of their resources.
Reboot both servers
Reboot physical server (VMWare server)
Restore a backup from 1 and 2 months ago of both servers to discard code and BD data errors
Review code (we do not touch code from before the error)
We did some queries from terminal's webserver and they were fast. We looked for the BD log and we could see that queries have been doing one by one in a "slow" velocity (0,5s for query aprox, depending on the query it can be more than a second).
We suspect that PHP is doing something bad, but we do not touch the code and we do not update PHP version or MySQL version. We want to try to update MySQL version in a new virtual machine and migrate all the data there, but we think it will not solve the problem.
The connection between servers is perfect and we think that layer 2 should not be the problem. In the same webserver we have another web application (Moodle) that connects to this BD server too, and it does not have this problem...
What can be the problem? It's very strange this change of behaviour of the web application. We were on holidays on August and we returned and we found this problem.
For more information we use PHP 5.6.40 (webserver) and MySQL Ver 14.14 Distrib 5.1.60, readline 5.1 (BD server). (yes, we know they are old versions but the web application and BD are old too)
I hope someone can help us, we are a little bit lost.
Thanks for your help!
I'd try updating your servers for one, issues like these are commonly caused by older software.
I'd also start logging or looking into loads so you can determine whatever causes the slow speeds you're getting.
I have a CentOS 7 server.
Since a few days the server behavior has been critically slow due a lot of mysql querys in the web app we have there.
I´ve tried to setup mariaDB server.cnf to log the slow-querys and the general-querys whitout having any response or log, I don´t know why it isn´t log any query.
Please help...
The setup for the logs is rather convoluted.
You also need
log_output = FILE
My company php web site is connected to an Advantage Database Server where are stored all necessary data such as users, passwords and customer registry.
Lately we started to get an error requesting web pages:
Warning: SQL error: [Extended Systems][Advantage SQL][ASA] Error 6303: Maximum Advantage Database Server connections exceeded. axServerConnect, SQL state HY000 in SQLDriverConnect in C:\...\www\... on line...
It's becoming critical day by day and it can happen once a week or twice a day without an apparent reason.
When website crashes, database service still working great with other applications connected and the only way to restore web service is to restart apache web server.
On database server we've got ads.ini configuration file in C:\Windows folder where we raised max connections setting with "MAX_CONNECTIONS=1000" which is really big compared to our needs.
Can it be useful if we set also "RETRY_ADS_CONNECTS = 1" ??
I found this post where R&D confirms a bug in may 2009:
Is this a bug with Advantage Database?
Has this been fixed? In wich release?
Where can i see the real number of connections open by apache on db?
Each php page closes ads connection on footer, what can cause connections to exceed??
Thanks in advance for help.
-
ENVIRONMENT INFO
Database:
Advantage Database Server 10.10.0.6 on Windows 2003 server
Web server:
Apache/2.0.59 (Win32) mod_ssl/2.0.59 OpenSSL/0.9.8d PHP/4.4.7 on Windows XP pro
On phpinfo() page we get "Advantage Version" "8.00.0.0".
Why this? Do we need to upgrade php-advantage extension?
Lots of questions, but I will try and address each.
1) 6303 Error. Using MAX_CONNECTIONS is the correct way to resolve this.
Make sure MAX_CONNECTIONS is in the [SETTINGS] section
Check if Apache / PHP / ADS driver is using the correct ads.ini file. You can use Process Monitor from Sysinternals to see what ads.ini file was opened successfully. If you upgrade your PHP driver you can set an environment variable adsini_path to point to the directory where the ads.ini lives.
2) Setting RETRY_ADS_CONNECTS=1 will be helpful. This also goes under the [SETTINGS] section of the ads.ini. When an ADS client receives a networking error (generally a 6000 class error) then the error is cached by the client driver and subsequent attempts to connect will use the cached error vs. retrying. Setting RETRY_ADS_CONNECTS will tell the ADS client to ignore the cached error and retry the connection
3) Bug: Looks like this was fixed in 9.10.0.9 version of the driver based on the release notes on http://devzone.advantagedatabase.com.
Fixed an issue where the garbage collection reference count on a
connection would be incorrect if multiple SQL statements were opened
on it.
Since you are running a 10.1 server you may look at updating to a 10.1 client which will also contain the fix. 10.1 Advantage PHP Driver
4)See the real number of connection
I would recommend using the stored procedure sp_mgGetConnectedUsers, you can use ARC (Advantage Data Architect) but it can be difficult to group, order, etc.
Since you are using 10.1 you can include the results of the stored procedure in a query such as
SELECT COUNT(*) FROM (EXECUTE PROCEDURE sp_mgGetConnectedUsers()) u WHERE ADDRESS='xxx.xxx.xxx.xxx'
You could also use other fields to identify the PHP application such as UserName (the server name), DictionaryUser (assuming the php application uses a unique user), ApplicationID
5) PHPINFO shows Advantage Version of the Advantage PHP client driver. You may want to upgrade the client driver for reasons mentioned above. Should be as simple as swapping DLL files (ace32.dll, axces32.dll, adsodbc.dll and php_advantage.dll), but I would recommend testing first to ensure you get everything.
I have two servers, both running CentOS 5.7 and cPanel-CURRENT. One is x86 and the other is x64. Both are using Apache 2.2.21, PHP 5.3.8 and MySQL 5.1.
If I query the local database on any one of the servers, the results are returned instantly. In this instance a few thousand results are being returned. However running the same query from one server to another and the query takes 10+ seconds to complete.
If I use MySQL Workbench 5.2 to query the remote database from my workstation with the same query, it completes in less than a second, which makes me think there's a problem with PHP or something else server-related.
Has anyone else encountered this issue before and know how to resolve it? Any help would be greatly appreciated.
First guess:
It might be a DNS issues and you may use --skip-name-resolve option in my.cnf or you may use only IP addresses in the MySQL grant tables.
Second guess:
It might be a problem with the security level and I suggest to temporarily disable selinux or the firewall and run the test again.
I had this exact issue with a PHP/MySQL application.
Just wanted to share that adding "skip-name-resolve" to my.ini fixed it for me. I was a little confused by this since my application on desktop 1 was accessing desktop 2 (mysql) using IP address. I am going between two Windows 7 desktops.
We have our database servers separate from our webserver. The database servers are replicated (we know there is overhead here). Even with replication turned off however, performance for large number of queries in a PHP script is 4 times slower than our staging server that has the db and apache on the same machine. I realize that network latency and other issues with a network mean that there is no way they will be equal, but our productions servers are exponentially more powerful and our production network is all on gigabit switches. We have tuned MYSQL as best as we can but the performance marker is still at 4x slower. We are running over nginx with Apache proxies and replicated MYSQL dbs. UCarp is also running. What are some suggestions for areas to look for improving the performance? I would be happy with twice as slow on production.
It's difficult to do much more than stab in the dark given your description, but here's some starting points to try independently, which will hopefully narrow down the cause:
Move your staging DB to another host
Add your staging host to the production pool and remove the others
Profile your PHP script to ensure it's the queries causing the delay
Use an individual MySQL server rather than via your load balancer
Measure a single query to the production pool and the staging server from the MySQL client
Run netperf between your web server and your DB cluster
Profile the web server with [gb]prof
Profile a MySQL server receiving the query with [gb]prof
If none of these illuminate anything other than the expected degradation due to the remote host, then please provide a reproducible test case and your full MySQL config (with sensitive data redacted.) That will help someone more skilled in MySQL assist you ;)
Not every web request on a web site will (if properly designed) need a mysql connection. Most likely, if you are requiring a connection on every http request, your application will not scale and will start having issues very quickly.
Do more caching at app. server to request mysql less often. E.g. use
memcache.
Try to use persistent connections from application to your mysql servers.
Use mysql data compression.
Minify data (limit your selects, use column names instead of "*" in select statements)
Shamanic tuning:
Make sure, that nothing slows down network at mysql servers: big firewall rulesets, network filters, etc.
Add another (client inaccesible) network interface for app. server
and mysql server.
Tune network connection between app. server and mysql. Sometimes you
can win several ms by creating hardcoded network routes.
Don't think any of above would help - if network connection is slow, nothing of above will significantly speed it up.