High CPU usage on server by MySQL on AWS ubuntu server - php

MySQL showing very high CPU usage. like over 200% sometime.
I had slow query log on. and slow query time set to 1s.
there is not any slow query.
I am using code igniter PHP.
my my.cnf file :
[mysqld]
innodb_io_capacity=2000
innodb_read_io_threads=32
innodb_write_io_threads=32
innodb_log_buffer_size=250M
innodb_thread_concurrency=0
innodb_buffer_pool_size=1000M
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
##
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
#
# * Fine Tuning
#
innodb_buffer_pool_size = 12G
key_buffer_size = 1G
max_allowed_packet = 256M
thread_stack = 192K
thread_cache_size = 100
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover-options = BACKUP
max_connections = 300
#innodb_buffer_pool_instances=8
#innodb_read_io_threads=8
#innodb_write_io_threads=8
#open_files_limit = 1024
#table_open_cache = 400
server_id = 2
bind-address = 0.0.0.0
log_bin = /var/log/mysql/mysql-bin.log
log_bin_index = /var/log/mysql/mysql-bin.log.index
relay_log = /var/log/mysql/mysql-relay-bin
relay_log_index = /var/log/mysql/mysql-relay-bin.index
expire_logs_days = 10
max_binlog_size = 100M
log_slave_updates = 1
auto-increment-increment = 2
auto-increment-offset = 2
replicate-ignore-db=phpmyadmin
replicate-ignore-db=mysql
skip-name-resolve
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit=0
query_cache_size=0
sort_buffer_size = 2M
join_buffer_size = 128K
read_buffer_size = 128K
read_rnd_buffer_size=256K
what can I do to reduce CPU usage?
Is there any way to find which queryes are causing high cpu usage o

there is not any slow query
There aren't any slow queries, or there aren't any slow queries showing in the log? I think your first prt of call would be to check if the query logging is working as you expect. This can be done easily using discretionary locks without adding additional load on the system.
Assuming that there really are no queries taking over 1 second, then the load arises fro lots of queries taking less than 1 second. You need to capture them all to identify and prioritize your tuning efforts. But you might start by enabling the query cache (which you have disabled for some reason).

Related

connection time out when trying to connect to remote database from app

I'm trying to connect to a remote database (It's a MySQL server on AWS EC2). Connection speed is okay when I try to connect using MySQL workbench or other DB management tools. But when I try to connect via my application (PHP/Laravel) the connection is too slow that it will time out.
Here is the result of dump(DB::connection());
I tried adding skip-external-locking and skip-name-resolve to the MySQL configuration file but it didn't change the result.
I tried running a MySQL query manually from my terminal to the database server with this command to determine how much it will take to run:
time mysql -u'username' -p'password' -e'show status'
The result was 0.05s user 0.04s system 0% cpu 10.582 total
I'll put the configuration file here, please let me know if you know a way to fix this issue:
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
skip-name-resolve
key_buffer_size = 256M
max_allowed_packet = 1M
thread_cache_size = 8
myisam-recover-options = BACKUP
innodb_buffer_pool_size = 1G
innodb_log_file_size=50M
innodb_flush_log_at_trx_commit=0
sync_binlog=0
innodb_flush_method=O_DIRECT
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M
innodb_file_per_table = 1
myisam_sort_buffer_size = 64M
read_rnd_buffer_size = 4M
read_buffer_size = 1M
sort_buffer_size = 1M
table_open_cache = 256
Also, someone with the same problem here solve it by commented out gethostbyaddr() but they didn't provide enough information.
UPDATE
When I use DB::select('select * from users limit 100'); in my laravel code, it's returning records from the database.
END OF UPDATE
Result of SHOW GLOBAL VARIABLES LIKE '%timeout%'; I'm in eastern Europe and the AWS EC2 server is in Ohio (east usa).
Result of SHOW GLOBAL VARIABLES LIKE '%connect%';
Result of SHOW GLOBAL STATUS LIKE '%connect%';
In your my.cnf [mysqld] section,
thread_cache_size=100 # from 8 for the cap suggested in 8.0 ref manual.
table_open_cache=3000 # from 256 to avoid table open thrashing
innodb_open_files=3000 # should always match table_open_cache
just a few details to consider with available information at this moment.
In your my.cnf [mysqld] section,
connect_timeout=30 # from 10 (seconds) since you are are many KM away from your host in USA-East.
Please share your code that 'connect's, processes, closes connections.
You appear to be leaving threads_connected when finished with a user's activities.

How to make faster queries on remote mysql server

I have 2 servers on OVH, they are on different datacenter in the Europe (SBG1 and GRA1). My controlled ping is 10ms.
My websites run many insert and read queries.
When I try my local mysql server it is very fast but when I use the remote mysql server the queries run is delayed.
My Remote Mysql Configuration
# Percona Server template configuration
[mysqld]
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
pid-file=/var/run/mysqld/mysqld.pid
skip-name-resolve
bind-address = 5.196.77.XXX
# skip-networking
sql-mode = 'NO_AUTO_CREATE_USER'
explicit_defaults_for_timestamp = 1
# MyISAM #
key_buffer_size = 2G
# SAFETY #
max_allowed_packet = 10G
# CACHES AND LIMITS #
tmp_table_size = 32M
max_heap_table_size = 32M
query_cache_type = 1
query_cache_size = 2M
query_cache_limit = 1M
join_buffer_size = 6M
max_connections = 600
thread_cache_size = 100
open_files_limit = 65535
table_definition_cache = 4096
table_open_cache = 4096
# INNODB #
innodb_flush_method = O_DIRECT
innodb_log_files_in_group = 2
innodb_log_file_size = 512M
innodb_flush_log_at_trx_commit = 1
innodb_file_per_table = 1
innodb_buffer_pool_size = 20G
innodb_data_file_path =ibdata1:20M:autoextend
# LOGGING #
log_error = /var/log/mysql/mysql-error.log
log-queries-not-using-indexes = 1
slow_query_log = 1
long_query_time = 3
slow_query_log_file = /var/log/mysql/mysql-slow.log
I don't know how can i connect local connection each servers.
Do i have to use vRack? (I have but i don't know have can i use it)
OR
Do i have to move my servers to same location?
What should i do
Replication enables data from one MySQL database server (the master)
to be copied to one or more MySQL database servers (the slaves).
Replication is asynchronous by default; slaves do not need to be
connected permanently to receive updates from the master.
If I understood you have a server S-A width database D-A and a server B with a database D-B and a local Server S-Local with database D-Local.
You can replicate the D-A and D-B databaseson your local Server S-Local. So your query will be faster.
I'm not sure if there is communication between S-A and S-B, but you can even replicate database D-A on Server S-B and database D-B on Server S-A.
My team has to query a distant server. It is very slow. With replication, our query are connected to a local replicated server and it is very useful

Mysql do not release memory

I have a project that uses 5 different data bases(oracle, mssql, ibm Informix , etc, ... and ...mysql)
Every timeout cron imports data into MySQL. Unfortunatelly we have one very huge cron script, that imports nearly 400 k rows. I made it to import data by small packs, but still, when cron runs this script, MySQL takes nearly 2 GB og memory, and wait for next call. Than takes 2 GB more, and this repeats before MySQL crashes.
We have already faced to the problem that MySQL innodb was corrupted and ran recovery mode from 1 to 4 to restart data base several times.
Total server virt memory is 12 GB.
Our configuration of /etc/my.cnf.d/ server.cnf looks like this:
#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#
# this is read by the standalone daemon and embedded servers
[server]
# this is only for the mysqld standalone daemon
[mysqld]
default_storage_engine=innodb
character-set-server=utf8
back_log = 50
max_connections = 100
max_connect_errors = 10
table_open_cache = 2048
max_allowed_packet = 64M
binlog_cache_size = 16M
max_heap_table_size = 1G
read_buffer_size = 8M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
thread_cache_size = 32
# You should try [number of CPUs]*(2..4) for thread_concurrency
thread_concurrency = 8
query_cache_size = 64M
query_cache_limit = 1M
tmp_table_size = 1G
thread_stack = 240K
transaction_isolation = REPEATABLE-READ
ft_min_word_len = 4
log-bin=mysql-bin
binlog_format=mixed
#slow_query_log
#long_query_time = 10
group_concat_max_len = 10000
#*** MyISAM Specific options
key_buffer_size = 16M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 64M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover
# *** INNODB Specific options ***
innodb_file_per_table
innodb_buffer_pool_size=8G
#innodb_data_file_path = ibdata1:10M:autoextend
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_log_file_size = 512M
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120
expire_logs_days=5
[mysqldump]
# Do not buffer the whole result set in memory before writing it to
# file. Required for dumping very large tables
quick
max_allowed_packet = 256M
[mysql]
no-auto-rehash
# Only allow UPDATEs and DELETEs that use keys.
safe-updates
[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M
[mysqlhotcopy]
interactive-timeout
[mysqld_safe]
# Increase the amount of open files allowed per process. Warning: Make
# sure you have set the global system limit high enough! The high value
# is required for a large number of opened tables
open-files-limit = 8192
If any info is necessary to find out why this happens I will provide it.
Please! help me to find out the solution ! Should I reinstall MySQL, or it is still possible to resolve by config modifications???

Mysql Config Optimization Suggestion

I have server with 8 GB RAM and 8 Core, Running one website, I have done mysql configuration setting, on which server working normally, but in some times (1 to 2 times in 24 hours), it takes so much load. Os running CentOS
Need suggestion.
[mysqld]
connection settings
max_connect_errors=10
max_connections=500
max_user_connections=700
wait_timeout=60
connect_timeout=10
interactive_timeout=60
innodb_buffer_pool_size=6GB
cache settings
query_cache_limit=128M
query_cache_size=10M
query_cache_type=1
table_open_cache=5000
thread_cache_size=250
buffer sizes
key_buffer=1288M
sort_buffer_size=20M
read_buffer_size=20M
join_buffer_size=20M
tmpdir / temp table sizes
tmp_table_size=256M
max_heap_table_size=256M
misc. settings
default-storage-engine = MYISAM
datadir=/var/lib/mysql
skip-external-locking
server-id = 1
open-files-limit = 96384
max_allowed_packet = 300M
innodb settings
innodb_data_file_path = ibdata1:10M:autoextend
innodb_thread_concurrency = 10
innodb_buffer_pool_size = 1000M
innodb_log_buffer_size = 500M
innodb_file_per_table = 1
[mysqld_safe]
open-files-limit = 16384
[mysqldump]
quick
max_allowed_packet=600M
[myisamchk]
key_buffer = 128M
sort_buffer = 128M
read_buffer = 128M
write_buffer = 16M
[mysql]
no-auto-rehash
Right now you have assigned more RAM to mysql than you have physically on server. Please check my comment below against variables-
[mysqld]
#connection setting
max_connect_errors=10
max_connections=500 #each connection consumes server resources, so keep how many required.
max_user_connections=700 #It should be always less than max_connections,even you should remove it.
wait_timeout=60
connect_timeout=10
interactive_timeout=60
innodb_buffer_pool_size=6GB #duplicate as already mentioned below
#cache setting
query_cache_limit=128M #this should be less than query_cache_size, 2M will be sufficient we should not cache heavy queries as it will slow the process.
query_cache_size=10M #can keep 20M or increase as per requirement.
query_cache_type=1
table_open_cache=5000
thread_cache_size=250 --even not impact much to performance but still should not more than 100.
#buffer sizes
key_buffer=1288M # keep it 20M as your server is innodb
sort_buffer_size=20M #this variable consumes server resources for each connection, 2M should be sufficient but can increase based on your queries.
read_buffer_size=20M #this variable consumes server resources for each connection, 2M should be sufficient but can increase based on your queries.
join_buffer_size=20M #this variable consumes server resources for each connection, 2M should be sufficient but can increase based on your queries.
#tmpdir / temp table sizes
tmp_table_size=256M
max_heap_table_size=256M
#misc. settings
default-storage-engine = MYISAM # change to innoDB
datadir=/var/lib/mysql
skip-external-locking
server-id = 1
open-files-limit = 96384
max_allowed_packet = 300M # You can reduce it to 64M or how much really required, fine if your application is sending large packet.
#innodb settings
innodb_data_file_path = ibdata1:10M:autoextend
innodb_thread_concurrency = 0 #mysql can utilize it better read this > http://dba.stackexchange.com/questions/5666/possible-to-make-mysql-use-more-than-one-core
innodb_buffer_pool_size = 1000M # It should be how much you can increase in innodb upto 80% of RAM.
innodb_log_buffer_size = 500M #8M is fine can increase as per server write in your environment but not 500M, as consumes server resources based on no of connections.
innodb_file_per_table = 1
[mysqld_safe]
log-error=/var/log/mysql/mysqld.log
[mysqldump]
quick
Note: You can leave other sections except [mysqld], [mysqld_Safe] nd [mysqldump]
Finally you can test your server with below configuration-
[mysqld]
#connection setting
max_connect_errors=10
max_connections=400
wait_timeout=60
connect_timeout=10
interactive_timeout=60
#cache setting
query_cache_limit=2M
query_cache_size=50M
query_cache_type=1
table_open_cache=5000
thread_cache_size=100
#buffer sizes
key_buffer_size=20M
sort_buffer_size=2M
read_buffer_size=2M
join_buffer_size=2M
#tmpdir / temp table sizes
tmp_table_size=256M
max_heap_table_size=256M
#misc. settings
default-storage-engine = innoDB
datadir=/var/lib/mysql
skip-external-locking
server-id = 1
open-files-limit = 65535
max_allowed_packet = 64M
#innodb settings
innodb_data_file_path = ibdata1:10M:autoextend
innodb_thread_concurrency = 0
innodb_buffer_pool_size = 4G
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
[mysqldump]
quick
Note: If you can reduce total_connections then you can increase innodb_buffer_pool_size and can get better performance. with 250 users you can increase innodb_buffer_pool_size from 4GB to 5GB.
Still first you need to test your server and can +/- as per requirement/performance.

Slow UPDATE statements in MySQL

I recently moved my website to another server and when i run a script that makes a lot of UPDATE statements i see very slowly results.
OLD_SERVER : Intel(R) Xeon(R) CPU E5-2650L 0 # 1.80GHz with 8 cores and 1.500 MB RAM
SERVER : Intel(R) Core(TM) i7-4770 CPU # 3.40GHz 8 cores and 32GB RAM + 2 ssd in RAID ( 10x times better then the old)
** php script code **
$startTime = microtime(true);
// change this with a mysql query
$update_result = mysql_query("
UPDATE some_table
SET order_id = ".$random_order_id."
WHERE id = ".$row_get['id']."
");
// Your content to test
$endTime = microtime(true);
$elapsed = $endTime - $startTime;
echo "Execution time : $elapsed seconds\n";
if(!$update_result)
{
return array(
"result" => false,
"code" => 502,
"reason" => "SQL Update error"
);
}
** OLD_SERVER script output (very fast): **
306497 will have order_id = 49438<br/>
Execution time : 0.00071907043457031 seconds
306505 will have order_id = 113556<br/>
Execution time : 0.00055885314941406 seconds
306508 will have order_id = 295573<br/>
Execution time : 0.00074100494384766 seconds
306511 will have order_id = 206028<br/>
Execution time : 0.00042295455932617 seconds
306518 will have order_id = 241993<br/>
Execution time : 0.00048589706420898 seconds
iotop renders 10MB/sec
** NEW_SERVER script output (very slow): **
10995 will have order_id = 94532<br/>
Execution time : 0.030339956283569 seconds
11021 will have order_id = 158848<br/>
Execution time : 0.060288906097412 seconds
11035 will have order_id = 288621<br/>
Execution time : 0.030526876449585 seconds
11059 will have order_id = 194945<br/>
Execution time : 0.031852960586548 seconds
11089 will have order_id = 176289<br/>
Execution time : 0.030807018280029 seconds
11102 will have order_id = 80207<br/>
Execution time : 0.059854984283447 seconds
11147 will have order_id = 33899<br/>
Execution time : 0.030609846115112 seconds
11392 will have order_id = 124314<br/>
Execution time : 0.031843900680542 seconds
11541 will have order_id = 249986<br/>
iotop renders 300KB/sec
** /etc/mysql/my.cnf **
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
#bind-address = 148.251.126.39
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 256M
# query_cache_size=0
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
innodb_log_file_size = 1G
innodb_io_capacity = 20000
innodb_read_io_threads = 5000
innodb_write_io_threads = 5000
Where is the problem ?? I don't understand.
Thanks.
Please check for the indexes first,
use
SHOW INDEXES FROM some_table;
on both the servers, and check whether there is any index on field 'id' in the old server.
Clearly you have upgraded your server configurations, so the issue must be in replication.

Categories