I have an AWS EC2 instance with DUAL-CORE and 4 GB Memory. I have setup my Apache2 HTTP server running PHP "7.0.30" and MySQL "Ver 14.14 Distrib 5.7.22".
There are various devices that are sending GET/POST request to my Http server. Each post and get request using select and update queries.
Right now, there are around 200 devices which are hitting my Http server simultaneously and hitting SQL queries of select and update together. These hits contain data in JSON formats.
The problem is that my MYSQL server has become too much slow. It takes long time to gather data from select queries and load pages.
From phpMyAdmin, I see a number of sleep processes in status for queries. I also have tuned various parameters of my SQL server but no result.
One of the major query that is taking time is update query which is updating long text data in table and is coming from device in every 60 seconds simultaneously and we see its processes empty after a long period of time in MYSQL server status.
Is there a way to optimize it using SQL parameters to keep MYSQL server fast even with 1000s of queries with multiple connections coming to update the table column having long text ?
Most of the Global variables are with default values. I also tried changing values of Various Global variables but it didn't produce any result.
How can I reduce this slow processing of queries?
P.S: I believe the issue is due to Update queries. I have tuned Select queries and they seems fine. But, for UPDATE queries, I see sleep of upto 12 seconds in Processes tab of phpMyAdmin.
I have added link to the image having this issue
(Here, you can see sleeps of even 13 seconds, all in UPDATE queries) :
Here is the PasteBin for the query of an UPDATE operation:
https://pastebin.com/kyUnkJmz
That is ~25KB for the JSON! (Maybe 22KB if backslashes vanish.) And 40 inserts/sec, but more every 2 minutes.
I would like to see SHOW CREATE TABLE, but I can still make some comments.
In InnoDB, that big a row will be stored 'off record'. That is, there will be an extra disk hit to write that big string elsewhere.
Compressing the JSON should shrink it to about 7K, which may lead to storing that big string inline, thereby cutting back some on the I/O. Do the compression in the client to help cut back on network traffic. And make the column a BLOB, not TEXT.
Spinning drives can handle about 100 I/Os per second.
The 200 devices every 5 seconds needs to average 40 writes/second in order to keep up. That's OK.
Every 2 minutes there are an extra 40 writes. This may (or may not) push the amount of I/O past what the disk can handle. This may be the proximate cause of the "updating for 13 seconds" you showed. That snapshot was taken shortly after a 2-minute boundary?
Or are the devices out of sync? That is do the POSTs come all at the same time, or are they spread out across the 2 minutes?
If each Update is a separate transaction (or you are running with autocommit=ON), then there is an extra write -- for transactional integrity. This can be turned off (tradeoff between speed and security): innodb_flush_log_at_trx_commit = 2. If you don't mind risking 1 second's worth of data,this may be a simple solution.
Is anything else going on with the table? Or is it just these Updates?
I hope you are using InnoDB (which is what my remarks above are directed toward), because MyISAM would be stumbling all over itself with fragmentation.
Long "Sleeps" are not an issue; long "Updates" are an issue.
More
Have an index on usermac so that the UPDATE does not have to slog through the entire table looking for the desired row. You could probably drop the id and add PRIMARY KEY(usermac).
Some of the comments above are off by a factor of 8 -- there seem to be 8 JSON columns in the table, hence 200KB/row.
Related
Why is connecting to my InnoDB database often delayed by an integer amount of seconds?
Some background
I have a very small MySQL database, consisting of a table 'users' (150 records) and a table 'connections' (growing to 150*150 records). Tables and indexes add up to less than 5 MB.
When users are active, 5-50 records in 'connections' are changed (the weight is changed) or added (if they didn't exist yet). The whole app runs smoothly and load times are below ~100 ms.
Except when they are not.
The details
Under even quite small loads, page load times spike from 60 ms to somewhere between 1,000 ms and 10,000 ms.
Using the profiler in Symfony, I could pin down the delay for 95+% on the 'getRepository' statement, while the queries took only ~1 ms per query. This led me to believe that connecting to the database was the slow action. I wrote a helper script regularly connecting to the database to test this theory.
<?php // call this script commandline using watch
$a = microtime(true);
$pdo = new PDO('mysql:host=127.0.0.1;dbname=mydb','myuser','mypass');
file_put_contents( 'performance.txt', (microtime(true)-$a).PHP_EOL, FILE_APPEND );
The mystery
Connecting to the database took consistently 1-3 ms, or 1,001-1,003 ms, or 2,001-2,003 ms, or 3,001-3,003 ms, etc. An integer amount of seconds, plus the normal time. Nothing in between, like 400 ms or 800 ms. With no writes going on, the connection was made almost instantly. As soon as some writes were performed via the app, the higher numbers were reached.
What is causing this behavior? The InnoDB page_cleaner appears to do its work every 1,000 ms, maybe that's part of the explanation?
More importantly, how can I fix this? I was thinking of switching to MEMORY tables, but I'd say more elegant options should be available.
EDIT
On request, the variables and global status.
Additional information: I connect directly to 127.0.0.1 (see the code snippet above) and I tested the skip-name-resolve flag to no effect. It's a Debian server, by the way.
EDIT 2
I found the delays were either 1, 3, 7 or 15 seconds. Notice the pattern: 1 second, +2s, +4s, +8s. This really looks as some timeout issue...
It's common that reverse dns lookup takes a long time. Along with the size of the host_cache it can give a erratic behaviour.
Turn it off by adding this to my.cnf
[mysqld]
skip-name-resolve
Note that all grants must be by ip, not by name, if you change this.
There is more to read in the manual
I'm puzzled; I assume a slow query.
Note: all my queries are tested and run great when there`s less people using my app/website (less then 0.01sec each).
So I've some high cpu usage with my current setup and I was wondering why? Is it possible it's an index issue?
Our possible solution: we thought we could use an XML cache file to store the informations each hour, and then reduce the load on our MySQL query? (update files each hour).
Will it be good for us to do such things? Since we have an SSD drive? Or will it be slower then before?
Currently in high traffic time, our website/app can take up to 30 seconds before return the first byte. My website is running under a Plesk 12 server.
UPDATE
Here's more informations about my mysql setup..
http://pastebin.com/KqvFYy8y
Is it possible it's an index issue?
Perhaps but not necessarily. You need first to identify which query is slow. You find that in the slow query log. Then analyze the query. This is explained in literature or you can contact a consultant / tutor for that.
We thought we could use an xml cache file to store the informations each hour.. and then reduce the load on our mysql query?
Well, cache invalidation is not the easiest thing to do, but with a fixed rythm every hour this seems easy enough. But take care that it will only help if the actual query you cache was slow. Mysql normally has a query cache built in, check if it is enabled or not first.
Will it be good for us to do such things?
Normally if the things to do are good, the results will be good, too. Sometimes even bad things will result in good results, so such a general question is hard to answer. Instead I suggest you gain more concrete information first before you continue to ask around. Sounds more like guessing. Stop guessing. Really, that's only for the first two minutes, after that, just stop guessing.
Since we have an ssd drive? Or will it be slower then before?
You can try to throw hardware on it. Again lierature and a consultant / tutor can help you greatly with that. But just stop guessing. Really.
I assume the query is not slow all the time. If this is true, the query is not very likely the problem.
You need to know what is using the CPU. Likely a runaway script with an infinite loop.
Try this:
<?php
header('Content-Type: text/plain; charset=utf-8');
echo system('ps auxww');
?>
This should return a list in this format:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
Scan down the %CPU column and look for your user name in the USER column
If you see a process taking 100% CPU, you may want to get the PID number and:
system('kill 1234');
Where 1234 is the PID
The mysql processes running at 441% and 218% seems very problematic.
Assuming this is a shared server, there may be another user running queries that is hogging the CPU. you may need to take that up with your provider.
I've been watching on one of my shared servers and the CPU for the mysql process has not gone over 16%.
MySQLTuner
From the link it appears you have heavy traffic.
The Tuner was running 23.5 minutes
Joins performed without indexes: 69863
69863 in 23.5 min. comes out to almost 50 queries per second.
Does this sound correct? Running a query with a JOIN 150 times per second.
Index JOIN Table
You have a query with a JOIN.
The tables are joined by column(s).
On the joined table add an index to the column that joins the two table together.
I'm really new to MySQL and just started to use the InnoDB table engine on my VPS.
Server info: 4 x 2.4 ghz, 8 Gb ram, 120gb Raid10.
My.cfg:
[mysqld]
innodb_file_per_table=1
innodb_buffer_pool_size=4G
innodb_log_file_size=512M
innodb_flush_log_at_trx_commit=2
Table to insert has 6 ints and 1 date and 1 trigger for mapping -> inserts 1 row for each row into a mapping table.
The last line helps a lot with speeding up the inserts (the database is 80% insert/update vs 20% read).
But when I do some tests, I call a PHP file in the webbrowser which will do 10 000 inserts, it takes about 3 seconds (is this fast/slow for this hardware?). But when I open multiple tabs and open the PHP file at the same time, they all have an execution time of 3 seconds, but they are waiting for each other to finish :/
Any ideas which settings I should add? Or other settings I should add for faster inserts are greatly appreciated!
3,300 inserts a second is quite respectable, especially with a trigger. It's hard to know more about that without understanding the tables.
What are you doing about commits? Are you doing all 10K inserts and then committing once? If so, other clients doing similar tasks are probably locked out until each client runs. That's a feature!
On the other hand, if you're using autocommit you're making your MySQL server churn.
The best strategy is to insert chunks of something like 100 or 500 rows, and then commit.
You should attempt to solve this kind of lockout not with configuration settings, but by tuning your web php application to manage transactions carefully.
You might want to consider using a MyISAM table if you don't need robust transaction semantics for this application. MyISAM handles large volumes of inserts more quickly. That is, if you don't particularly care if you read data that's current or a few seconds old, then MyISAM will serve you well.
We are connecting to an SQL Server 2008 on another box using PHP 5.3 and pdo_dblib.
Periodically (10+ times a day randomly throughout the day) we experience 3 minute periods where all SQL Select queries will take 21.01 seconds to run, regardless of the query. This is timed from within PHP (before and after DB query).
The SELECT statements range from complex joins with optimization to statements with single tables and explicit indices.
If, during a PHP request, we perform three SELECT statements before closing the DB connection, the total time would be 63.03 seconds (21.01 * 3). The same applies for 2 statements (42.02 seconds).
Using SQL Server tools, we've tracked the actual execution times of these scripts to be between 0.01 to 0.45 seconds, but these differences don't seem to reflect in the overall delay (it is always fixed at 21.01, not 21.46, etc).
We did seem to find some correlation of WinHTTP Proxy triggering at the start of these delayed periods but we have since disabled WinHTTP Proxy with no resolution.
Any advice?
When you look at your execution plan for these, what activity is taking the largest percentage of the resources? I'd specifically be on the lookout for Index Scans or Key/Bookmark/RID lookups. I'd also look at the lines connecting each activity and see how many records are being applied. Lastly, run the query using an Actual Query plan rather than an estimated one then look at the Actual vs Estimated rows returned for each action. They should be equal. If they are not, that's a good indication that your statistics are off.
Using a PHP script I need to update a number every 5 seconds while somebody is on my page. So let's say I have 300 visitors, each one spending about 1 minute on the page and every 5 seconds they stay on the page the number will be changed...which is a total of 3600 changes per minute. I would prefer to update the number in my MySQL database, except I'm not sure if it's not too inefficient to have so many MySQL connections (just for the one number change), when I could just change the number in a file.
P.S.: I have no idea weather 3600 connections/minute is a high number or not, but what about this case in general, considering an even higher number of visitors. What is the most efficient way to do this?
Doing 3,600 reads and writes per minute against the same file is just out of question. It's complicate (you need to be extremely careful with file locking), it's going to have an awful performance and sooner or later your data will get corrupted.
DBMSs like MySQL are designed for concurrent access. If they can't cope with your load, a file won't do it better.
It will fail eventually if the user count grows but the performance depends of your server setup and other tasks that are related to this update.
You can do a slight test and open up 300 persistent connections to your database end fire up as much query's you can in minute.
If you don't need it to be transactional (the order of executed query's is not important) then i suggest you to use memcached (or redis if you need to save stuff on disk) for this instead
If you save to file, you have to solve concurrency issues (and all but the currently reading/writing process will have to wait). The db solves this for you. For better performance you could use memcached.
Maybe you could do without this "do every 5s for each user" by another means (e.g. saving current time and subtracting next time the user does something). This depends on your real problem.
Don't even think about trying to handle this with files - its just not going to work unless you build a lock queue manager - and if you're going to all that trouble you might as well use the daemon to manage the value rather than just queue locks.
Using a DBMS is the simplest approach.
For a more efficient but massively more esoteric approach, write a single-threaded socket server daemon and have the clients connect to that. (there's a lib here for doing the socket handling, and there's a PEAR class for running PHP as a daemon)
files aren't transactional and you don't want to lose count so the database is the way to go
memcached's inc command is faster then the database and was the basis of i think one really fast view counting setup
if you use say a key per hour and switch so when a page view happens inc page:time occurs and you can have a process in the background collect the counts from the past hour and insert them in a database if the memcache fails you might lose the count for that hour but you will not have double counted or missed any and keeping counts per period gives interesting statistics
Using a dedicated temporary file will certainly be the most efficient disk access you can have. However, you will not be protected from concurrent access to the file in case your server uses multiple threads or processes. If what you want to do is update 1 number per user, then using a $_SESSION sub-variable will work, and I believe this is stored in memory, so it shouldbe very efficient. Then you can easily store this number into your database every 5 minutes per user