How to make faster queries on remote mysql server - php

I have 2 servers on OVH, they are on different datacenter in the Europe (SBG1 and GRA1). My controlled ping is 10ms.
My websites run many insert and read queries.
When I try my local mysql server it is very fast but when I use the remote mysql server the queries run is delayed.
My Remote Mysql Configuration
# Percona Server template configuration
[mysqld]
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
pid-file=/var/run/mysqld/mysqld.pid
skip-name-resolve
bind-address = 5.196.77.XXX
# skip-networking
sql-mode = 'NO_AUTO_CREATE_USER'
explicit_defaults_for_timestamp = 1
# MyISAM #
key_buffer_size = 2G
# SAFETY #
max_allowed_packet = 10G
# CACHES AND LIMITS #
tmp_table_size = 32M
max_heap_table_size = 32M
query_cache_type = 1
query_cache_size = 2M
query_cache_limit = 1M
join_buffer_size = 6M
max_connections = 600
thread_cache_size = 100
open_files_limit = 65535
table_definition_cache = 4096
table_open_cache = 4096
# INNODB #
innodb_flush_method = O_DIRECT
innodb_log_files_in_group = 2
innodb_log_file_size = 512M
innodb_flush_log_at_trx_commit = 1
innodb_file_per_table = 1
innodb_buffer_pool_size = 20G
innodb_data_file_path =ibdata1:20M:autoextend
# LOGGING #
log_error = /var/log/mysql/mysql-error.log
log-queries-not-using-indexes = 1
slow_query_log = 1
long_query_time = 3
slow_query_log_file = /var/log/mysql/mysql-slow.log
I don't know how can i connect local connection each servers.
Do i have to use vRack? (I have but i don't know have can i use it)
OR
Do i have to move my servers to same location?
What should i do

Replication enables data from one MySQL database server (the master)
to be copied to one or more MySQL database servers (the slaves).
Replication is asynchronous by default; slaves do not need to be
connected permanently to receive updates from the master.
If I understood you have a server S-A width database D-A and a server B with a database D-B and a local Server S-Local with database D-Local.
You can replicate the D-A and D-B databaseson your local Server S-Local. So your query will be faster.
I'm not sure if there is communication between S-A and S-B, but you can even replicate database D-A on Server S-B and database D-B on Server S-A.
My team has to query a distant server. It is very slow. With replication, our query are connected to a local replicated server and it is very useful

Related

connection time out when trying to connect to remote database from app

I'm trying to connect to a remote database (It's a MySQL server on AWS EC2). Connection speed is okay when I try to connect using MySQL workbench or other DB management tools. But when I try to connect via my application (PHP/Laravel) the connection is too slow that it will time out.
Here is the result of dump(DB::connection());
I tried adding skip-external-locking and skip-name-resolve to the MySQL configuration file but it didn't change the result.
I tried running a MySQL query manually from my terminal to the database server with this command to determine how much it will take to run:
time mysql -u'username' -p'password' -e'show status'
The result was 0.05s user 0.04s system 0% cpu 10.582 total
I'll put the configuration file here, please let me know if you know a way to fix this issue:
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
skip-name-resolve
key_buffer_size = 256M
max_allowed_packet = 1M
thread_cache_size = 8
myisam-recover-options = BACKUP
innodb_buffer_pool_size = 1G
innodb_log_file_size=50M
innodb_flush_log_at_trx_commit=0
sync_binlog=0
innodb_flush_method=O_DIRECT
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M
innodb_file_per_table = 1
myisam_sort_buffer_size = 64M
read_rnd_buffer_size = 4M
read_buffer_size = 1M
sort_buffer_size = 1M
table_open_cache = 256
Also, someone with the same problem here solve it by commented out gethostbyaddr() but they didn't provide enough information.
UPDATE
When I use DB::select('select * from users limit 100'); in my laravel code, it's returning records from the database.
END OF UPDATE
Result of SHOW GLOBAL VARIABLES LIKE '%timeout%'; I'm in eastern Europe and the AWS EC2 server is in Ohio (east usa).
Result of SHOW GLOBAL VARIABLES LIKE '%connect%';
Result of SHOW GLOBAL STATUS LIKE '%connect%';
In your my.cnf [mysqld] section,
thread_cache_size=100 # from 8 for the cap suggested in 8.0 ref manual.
table_open_cache=3000 # from 256 to avoid table open thrashing
innodb_open_files=3000 # should always match table_open_cache
just a few details to consider with available information at this moment.
In your my.cnf [mysqld] section,
connect_timeout=30 # from 10 (seconds) since you are are many KM away from your host in USA-East.
Please share your code that 'connect's, processes, closes connections.
You appear to be leaving threads_connected when finished with a user's activities.

High CPU usage on server by MySQL on AWS ubuntu server

MySQL showing very high CPU usage. like over 200% sometime.
I had slow query log on. and slow query time set to 1s.
there is not any slow query.
I am using code igniter PHP.
my my.cnf file :
[mysqld]
innodb_io_capacity=2000
innodb_read_io_threads=32
innodb_write_io_threads=32
innodb_log_buffer_size=250M
innodb_thread_concurrency=0
innodb_buffer_pool_size=1000M
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
##
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
#
# * Fine Tuning
#
innodb_buffer_pool_size = 12G
key_buffer_size = 1G
max_allowed_packet = 256M
thread_stack = 192K
thread_cache_size = 100
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover-options = BACKUP
max_connections = 300
#innodb_buffer_pool_instances=8
#innodb_read_io_threads=8
#innodb_write_io_threads=8
#open_files_limit = 1024
#table_open_cache = 400
server_id = 2
bind-address = 0.0.0.0
log_bin = /var/log/mysql/mysql-bin.log
log_bin_index = /var/log/mysql/mysql-bin.log.index
relay_log = /var/log/mysql/mysql-relay-bin
relay_log_index = /var/log/mysql/mysql-relay-bin.index
expire_logs_days = 10
max_binlog_size = 100M
log_slave_updates = 1
auto-increment-increment = 2
auto-increment-offset = 2
replicate-ignore-db=phpmyadmin
replicate-ignore-db=mysql
skip-name-resolve
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit=0
query_cache_size=0
sort_buffer_size = 2M
join_buffer_size = 128K
read_buffer_size = 128K
read_rnd_buffer_size=256K
what can I do to reduce CPU usage?
Is there any way to find which queryes are causing high cpu usage o
there is not any slow query
There aren't any slow queries, or there aren't any slow queries showing in the log? I think your first prt of call would be to check if the query logging is working as you expect. This can be done easily using discretionary locks without adding additional load on the system.
Assuming that there really are no queries taking over 1 second, then the load arises fro lots of queries taking less than 1 second. You need to capture them all to identify and prioritize your tuning efforts. But you might start by enabling the query cache (which you have disabled for some reason).

Xampp - MySql shutdown - Not working

I updated the MySQL version the correct way, however when clicking to start running it from the error and does not start.
Now I followed following points to install the same:
Install MySQL to C:\TEMP.
Make old installation folder to mysql_old.
copy the following folders "bin, include, lib, share, support-files" to xamp\mysql\ folder. I didn't copy the data folder
Copied my.ini file from old installation to new installation in xamp\mysql\bin\ folder
Copied the old data folder to new MySQL folder
Report Xampp
11:53:07 [mysql] Error: MySQL shutdown unexpectedly.
11:53:07 [mysql] This may be due to a blocked port, missing dependencies,
11:53:07 [mysql] improper privileges, a crash, or a shutdown by another method.
11:53:07 [mysql] Press the Logs button to view error logs and check
11:53:07 [mysql] the Windows Event Viewer for more clues
11:53:07 [mysql] If you need more help, copy and post this
11:53:07 [mysql] entire log window on the forums
Log MySQL Error:
2017-12-15T13:53:02.700418Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-12-15T13:53:02.700418Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2017-12-15T13:53:02.700418Z 0 [Note] c:\xampp\mysql\bin\mysqld.exe (mysqld 5.7.20) starting as process 7764 ..
my.ini
# The following options will be passed to all MySQL clients
[client]
# password = your_password
port = 3306
socket = "C:/xampp/mysql/mysql.sock"
# The MySQL server
[mysqld]
port= 3306
socket = "C:/xampp/mysql/mysql.sock"
basedir = "C:/xampp/mysql"
tmpdir = "C:/xampp/tmp"
datadir = "C:/xampp/mysql/data"
pid_file = "mysql.pid"
# enable-named-pipe
key_buffer = 16M
max_allowed_packet = 1M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
log_error = "mysql_error.log"
# Change here for bind listening
# bind-address="127.0.0.1"
# bind-address = ::1 # for ipv6
# Where do all the plugins live
plugin_dir = "C:/xampp/mysql/lib/plugin/"
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
# commented in by lampp security
#skip-networking
skip-federated
# Replication Master Server (default)
# binary logging is required for replication
# log-bin deactivated by default since XAMPP 1.4.11
#log-bin=mysql-bin
# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1
# Comment the following if you are using InnoDB tables
#skip-innodb
innodb_data_home_dir = "C:/xampp/mysql/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp/mysql/data"
#innodb_log_arch_dir = "C:/xampp/mysql/data"
## You can set .._buffer_pool_size up to 50 - 80 %
## of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
## Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
## UTF 8 Settings
#init-connect=\'SET NAMES utf8\'
#collation_server=utf8_unicode_ci
#character_set_server=utf8
#skip-character-set-client-handshake
#character_sets-dir="C:/xampp/mysql/share/charsets"
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
[isamchk]
#key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
Maybe try to start your mysql on diffrent port for example 3307

Mysql Config Optimization Suggestion

I have server with 8 GB RAM and 8 Core, Running one website, I have done mysql configuration setting, on which server working normally, but in some times (1 to 2 times in 24 hours), it takes so much load. Os running CentOS
Need suggestion.
[mysqld]
connection settings
max_connect_errors=10
max_connections=500
max_user_connections=700
wait_timeout=60
connect_timeout=10
interactive_timeout=60
innodb_buffer_pool_size=6GB
cache settings
query_cache_limit=128M
query_cache_size=10M
query_cache_type=1
table_open_cache=5000
thread_cache_size=250
buffer sizes
key_buffer=1288M
sort_buffer_size=20M
read_buffer_size=20M
join_buffer_size=20M
tmpdir / temp table sizes
tmp_table_size=256M
max_heap_table_size=256M
misc. settings
default-storage-engine = MYISAM
datadir=/var/lib/mysql
skip-external-locking
server-id = 1
open-files-limit = 96384
max_allowed_packet = 300M
innodb settings
innodb_data_file_path = ibdata1:10M:autoextend
innodb_thread_concurrency = 10
innodb_buffer_pool_size = 1000M
innodb_log_buffer_size = 500M
innodb_file_per_table = 1
[mysqld_safe]
open-files-limit = 16384
[mysqldump]
quick
max_allowed_packet=600M
[myisamchk]
key_buffer = 128M
sort_buffer = 128M
read_buffer = 128M
write_buffer = 16M
[mysql]
no-auto-rehash
Right now you have assigned more RAM to mysql than you have physically on server. Please check my comment below against variables-
[mysqld]
#connection setting
max_connect_errors=10
max_connections=500 #each connection consumes server resources, so keep how many required.
max_user_connections=700 #It should be always less than max_connections,even you should remove it.
wait_timeout=60
connect_timeout=10
interactive_timeout=60
innodb_buffer_pool_size=6GB #duplicate as already mentioned below
#cache setting
query_cache_limit=128M #this should be less than query_cache_size, 2M will be sufficient we should not cache heavy queries as it will slow the process.
query_cache_size=10M #can keep 20M or increase as per requirement.
query_cache_type=1
table_open_cache=5000
thread_cache_size=250 --even not impact much to performance but still should not more than 100.
#buffer sizes
key_buffer=1288M # keep it 20M as your server is innodb
sort_buffer_size=20M #this variable consumes server resources for each connection, 2M should be sufficient but can increase based on your queries.
read_buffer_size=20M #this variable consumes server resources for each connection, 2M should be sufficient but can increase based on your queries.
join_buffer_size=20M #this variable consumes server resources for each connection, 2M should be sufficient but can increase based on your queries.
#tmpdir / temp table sizes
tmp_table_size=256M
max_heap_table_size=256M
#misc. settings
default-storage-engine = MYISAM # change to innoDB
datadir=/var/lib/mysql
skip-external-locking
server-id = 1
open-files-limit = 96384
max_allowed_packet = 300M # You can reduce it to 64M or how much really required, fine if your application is sending large packet.
#innodb settings
innodb_data_file_path = ibdata1:10M:autoextend
innodb_thread_concurrency = 0 #mysql can utilize it better read this > http://dba.stackexchange.com/questions/5666/possible-to-make-mysql-use-more-than-one-core
innodb_buffer_pool_size = 1000M # It should be how much you can increase in innodb upto 80% of RAM.
innodb_log_buffer_size = 500M #8M is fine can increase as per server write in your environment but not 500M, as consumes server resources based on no of connections.
innodb_file_per_table = 1
[mysqld_safe]
log-error=/var/log/mysql/mysqld.log
[mysqldump]
quick
Note: You can leave other sections except [mysqld], [mysqld_Safe] nd [mysqldump]
Finally you can test your server with below configuration-
[mysqld]
#connection setting
max_connect_errors=10
max_connections=400
wait_timeout=60
connect_timeout=10
interactive_timeout=60
#cache setting
query_cache_limit=2M
query_cache_size=50M
query_cache_type=1
table_open_cache=5000
thread_cache_size=100
#buffer sizes
key_buffer_size=20M
sort_buffer_size=2M
read_buffer_size=2M
join_buffer_size=2M
#tmpdir / temp table sizes
tmp_table_size=256M
max_heap_table_size=256M
#misc. settings
default-storage-engine = innoDB
datadir=/var/lib/mysql
skip-external-locking
server-id = 1
open-files-limit = 65535
max_allowed_packet = 64M
#innodb settings
innodb_data_file_path = ibdata1:10M:autoextend
innodb_thread_concurrency = 0
innodb_buffer_pool_size = 4G
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
[mysqldump]
quick
Note: If you can reduce total_connections then you can increase innodb_buffer_pool_size and can get better performance. with 250 users you can increase innodb_buffer_pool_size from 4GB to 5GB.
Still first you need to test your server and can +/- as per requirement/performance.

MySQL Too Many Connections while under VMWare

After doing a lot of research and work, I am perplexed at this time on why my MySQL server keeps refusing connections. I have gone though many others who say to increase the max-connections, decreased the wait_timeout, and have been increasing the virtual machine's memory to 7 GB now, reserving 1.5 GB of memory in vCenter.
I am running this on a HP Prolaint DL 360 G5 with two Intel Xeon E5405 and 16 GB of RAM running vSphere 5.5 with vCenter. The MySQL VM is running Ubuntu 12.04 on a Virtual Machine Minimal install, having:
Active: 215 MB
Granted: 3.65 GB
Consumed: 1.66 GB
Balloon: 2.88 GB
CPU Maximum MHz: 344
CPU Avg MHz: 55.561
Disk Usage Maximum (KBps): 10803
Disk Highest Latency: 46
Disk Average Latency: 2
ulimit -n: 1024
Amount of connections while refusing connection: 16
I have apache with php, two openfire instances, teamspeak, and rsyslog connected to the database server. I have checked and none are using persistant connections.
Any idea why this happens every few days?
The my.cnf file:
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /mysql/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 192.168.2.7
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
wait_timeout = 360
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
max_connections = 1000
table_cache = 2000
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 20M
query_cache_size = 32M
#
# * InnoDB Configuration - gac64k56
#
innodb_buffer_pool_size = 2G
join_buffer_size = 256K
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

Categories