I'm trying to connect to a remote database (It's a MySQL server on AWS EC2). Connection speed is okay when I try to connect using MySQL workbench or other DB management tools. But when I try to connect via my application (PHP/Laravel) the connection is too slow that it will time out.
Here is the result of dump(DB::connection());
I tried adding skip-external-locking and skip-name-resolve to the MySQL configuration file but it didn't change the result.
I tried running a MySQL query manually from my terminal to the database server with this command to determine how much it will take to run:
time mysql -u'username' -p'password' -e'show status'
The result was 0.05s user 0.04s system 0% cpu 10.582 total
I'll put the configuration file here, please let me know if you know a way to fix this issue:
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
skip-name-resolve
key_buffer_size = 256M
max_allowed_packet = 1M
thread_cache_size = 8
myisam-recover-options = BACKUP
innodb_buffer_pool_size = 1G
innodb_log_file_size=50M
innodb_flush_log_at_trx_commit=0
sync_binlog=0
innodb_flush_method=O_DIRECT
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M
innodb_file_per_table = 1
myisam_sort_buffer_size = 64M
read_rnd_buffer_size = 4M
read_buffer_size = 1M
sort_buffer_size = 1M
table_open_cache = 256
Also, someone with the same problem here solve it by commented out gethostbyaddr() but they didn't provide enough information.
UPDATE
When I use DB::select('select * from users limit 100'); in my laravel code, it's returning records from the database.
END OF UPDATE
Result of SHOW GLOBAL VARIABLES LIKE '%timeout%'; I'm in eastern Europe and the AWS EC2 server is in Ohio (east usa).
Result of SHOW GLOBAL VARIABLES LIKE '%connect%';
Result of SHOW GLOBAL STATUS LIKE '%connect%';
In your my.cnf [mysqld] section,
thread_cache_size=100 # from 8 for the cap suggested in 8.0 ref manual.
table_open_cache=3000 # from 256 to avoid table open thrashing
innodb_open_files=3000 # should always match table_open_cache
just a few details to consider with available information at this moment.
In your my.cnf [mysqld] section,
connect_timeout=30 # from 10 (seconds) since you are are many KM away from your host in USA-East.
Please share your code that 'connect's, processes, closes connections.
You appear to be leaving threads_connected when finished with a user's activities.
Related
MySQL showing very high CPU usage. like over 200% sometime.
I had slow query log on. and slow query time set to 1s.
there is not any slow query.
I am using code igniter PHP.
my my.cnf file :
[mysqld]
innodb_io_capacity=2000
innodb_read_io_threads=32
innodb_write_io_threads=32
innodb_log_buffer_size=250M
innodb_thread_concurrency=0
innodb_buffer_pool_size=1000M
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
##
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
#
# * Fine Tuning
#
innodb_buffer_pool_size = 12G
key_buffer_size = 1G
max_allowed_packet = 256M
thread_stack = 192K
thread_cache_size = 100
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover-options = BACKUP
max_connections = 300
#innodb_buffer_pool_instances=8
#innodb_read_io_threads=8
#innodb_write_io_threads=8
#open_files_limit = 1024
#table_open_cache = 400
server_id = 2
bind-address = 0.0.0.0
log_bin = /var/log/mysql/mysql-bin.log
log_bin_index = /var/log/mysql/mysql-bin.log.index
relay_log = /var/log/mysql/mysql-relay-bin
relay_log_index = /var/log/mysql/mysql-relay-bin.index
expire_logs_days = 10
max_binlog_size = 100M
log_slave_updates = 1
auto-increment-increment = 2
auto-increment-offset = 2
replicate-ignore-db=phpmyadmin
replicate-ignore-db=mysql
skip-name-resolve
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit=0
query_cache_size=0
sort_buffer_size = 2M
join_buffer_size = 128K
read_buffer_size = 128K
read_rnd_buffer_size=256K
what can I do to reduce CPU usage?
Is there any way to find which queryes are causing high cpu usage o
there is not any slow query
There aren't any slow queries, or there aren't any slow queries showing in the log? I think your first prt of call would be to check if the query logging is working as you expect. This can be done easily using discretionary locks without adding additional load on the system.
Assuming that there really are no queries taking over 1 second, then the load arises fro lots of queries taking less than 1 second. You need to capture them all to identify and prioritize your tuning efforts. But you might start by enabling the query cache (which you have disabled for some reason).
I updated the MySQL version the correct way, however when clicking to start running it from the error and does not start.
Now I followed following points to install the same:
Install MySQL to C:\TEMP.
Make old installation folder to mysql_old.
copy the following folders "bin, include, lib, share, support-files" to xamp\mysql\ folder. I didn't copy the data folder
Copied my.ini file from old installation to new installation in xamp\mysql\bin\ folder
Copied the old data folder to new MySQL folder
Report Xampp
11:53:07 [mysql] Error: MySQL shutdown unexpectedly.
11:53:07 [mysql] This may be due to a blocked port, missing dependencies,
11:53:07 [mysql] improper privileges, a crash, or a shutdown by another method.
11:53:07 [mysql] Press the Logs button to view error logs and check
11:53:07 [mysql] the Windows Event Viewer for more clues
11:53:07 [mysql] If you need more help, copy and post this
11:53:07 [mysql] entire log window on the forums
Log MySQL Error:
2017-12-15T13:53:02.700418Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-12-15T13:53:02.700418Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2017-12-15T13:53:02.700418Z 0 [Note] c:\xampp\mysql\bin\mysqld.exe (mysqld 5.7.20) starting as process 7764 ..
my.ini
# The following options will be passed to all MySQL clients
[client]
# password = your_password
port = 3306
socket = "C:/xampp/mysql/mysql.sock"
# The MySQL server
[mysqld]
port= 3306
socket = "C:/xampp/mysql/mysql.sock"
basedir = "C:/xampp/mysql"
tmpdir = "C:/xampp/tmp"
datadir = "C:/xampp/mysql/data"
pid_file = "mysql.pid"
# enable-named-pipe
key_buffer = 16M
max_allowed_packet = 1M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
log_error = "mysql_error.log"
# Change here for bind listening
# bind-address="127.0.0.1"
# bind-address = ::1 # for ipv6
# Where do all the plugins live
plugin_dir = "C:/xampp/mysql/lib/plugin/"
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
# commented in by lampp security
#skip-networking
skip-federated
# Replication Master Server (default)
# binary logging is required for replication
# log-bin deactivated by default since XAMPP 1.4.11
#log-bin=mysql-bin
# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1
# Comment the following if you are using InnoDB tables
#skip-innodb
innodb_data_home_dir = "C:/xampp/mysql/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp/mysql/data"
#innodb_log_arch_dir = "C:/xampp/mysql/data"
## You can set .._buffer_pool_size up to 50 - 80 %
## of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
## Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
## UTF 8 Settings
#init-connect=\'SET NAMES utf8\'
#collation_server=utf8_unicode_ci
#character_set_server=utf8
#skip-character-set-client-handshake
#character_sets-dir="C:/xampp/mysql/share/charsets"
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
[isamchk]
#key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
Maybe try to start your mysql on diffrent port for example 3307
I have 2 servers on OVH, they are on different datacenter in the Europe (SBG1 and GRA1). My controlled ping is 10ms.
My websites run many insert and read queries.
When I try my local mysql server it is very fast but when I use the remote mysql server the queries run is delayed.
My Remote Mysql Configuration
# Percona Server template configuration
[mysqld]
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
pid-file=/var/run/mysqld/mysqld.pid
skip-name-resolve
bind-address = 5.196.77.XXX
# skip-networking
sql-mode = 'NO_AUTO_CREATE_USER'
explicit_defaults_for_timestamp = 1
# MyISAM #
key_buffer_size = 2G
# SAFETY #
max_allowed_packet = 10G
# CACHES AND LIMITS #
tmp_table_size = 32M
max_heap_table_size = 32M
query_cache_type = 1
query_cache_size = 2M
query_cache_limit = 1M
join_buffer_size = 6M
max_connections = 600
thread_cache_size = 100
open_files_limit = 65535
table_definition_cache = 4096
table_open_cache = 4096
# INNODB #
innodb_flush_method = O_DIRECT
innodb_log_files_in_group = 2
innodb_log_file_size = 512M
innodb_flush_log_at_trx_commit = 1
innodb_file_per_table = 1
innodb_buffer_pool_size = 20G
innodb_data_file_path =ibdata1:20M:autoextend
# LOGGING #
log_error = /var/log/mysql/mysql-error.log
log-queries-not-using-indexes = 1
slow_query_log = 1
long_query_time = 3
slow_query_log_file = /var/log/mysql/mysql-slow.log
I don't know how can i connect local connection each servers.
Do i have to use vRack? (I have but i don't know have can i use it)
OR
Do i have to move my servers to same location?
What should i do
Replication enables data from one MySQL database server (the master)
to be copied to one or more MySQL database servers (the slaves).
Replication is asynchronous by default; slaves do not need to be
connected permanently to receive updates from the master.
If I understood you have a server S-A width database D-A and a server B with a database D-B and a local Server S-Local with database D-Local.
You can replicate the D-A and D-B databaseson your local Server S-Local. So your query will be faster.
I'm not sure if there is communication between S-A and S-B, but you can even replicate database D-A on Server S-B and database D-B on Server S-A.
My team has to query a distant server. It is very slow. With replication, our query are connected to a local replicated server and it is very useful
I manage a file sharing website (like mediafire,hotFile, etc.) and there is a problem when people try to download from one of the servers.
It goes like this: I have one main server - there I have the mysql database and the website himself, and I have more servers to host the files.
The download process goes like this: user get a link to the outside server. This page connect remotely to the main server and make querys with the database. After the querys, the download get started.
Now, all of the downloads works fine, except from one of the servers. The weirdest thing is that sometimes the download from this server is works, and sometimes not!
It's about 70% of times the download from the problematic server is works, and 30% of times the download isn't works.
When the download isn't working the error message is:
Connect failed: Lost connection to MySQL server at 'reading authorization packet', system error: 0
The my.cnf looks like this:
[mysqld]
skip-name-resolve
bulk_insert_buffer_size = 8M
concurrent_insert = 2
connect_timeout = 10
default-storage-engine = MyISAM
innodb_buffer_pool_size=16M
interactive_timeout = 35
join_buffer_size = 2M
key_buffer_size = 192M
local-infile=0
log-error=/var/log/mysql/error.log
log-slow-queries
log-slow-queries=/var/log/mysql/mysql-slow.log
long_query_time=1
max_allowed_packet = 32M
max_connections = 3000
max_heap_table_size = 256M
max_user_connections= 400
max_write_lock_count = 8
myisam_max_sort_file_size = 256M
myisam_sort_buffer_size = 64M
open_files_limit=128K
query_alloc_block_size = 65536
query_cache_limit = 16M
query_cache_size = 128M
query_cache_type = 1
query_prealloc_size = 262144
range_alloc_block_size = 4096
read_buffer_size = 2M
read_rnd_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 48K
thread_cache_size = 512
tmp_table_size = 256M
transaction_alloc_block_size = 4096
transaction_prealloc_size = 4096
wait_timeout = 100
max_connect_errors = 5000
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
How can I fix the problem?
Thank you very much, and sorry for my English.
I have to update 11M rows from a database in a PHP script.
After some time, the script freezes or crashes. I have to restart EasyPHP 12, and reload it.
My configuration:
Windows 7 Pro 64 bits
Intel Core i7 860 2.8Ghz
8G RAM
My my.cnf file :
port = 3306
socket = /tmp/mysql.sock
[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 384M
max_allowed_packet = 1M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 32M
thread_concurrency = 8
log-bin=mysql-bin
server-id = 1
[mysqldump]
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
Here is the pseudo code that crash.
`For i to 100000 { do magic (check content on the web); UPDATE table; }`
You have to look into php.ini, not my.cnf.
I suppose you are performing some logic on a single record and updating that record, then on to the next one. In this case the update of a single record (or a subset of records) should not take that long.
The freeze or crash is either because your script hits the memory limit or its execution time limit.
You are probably running into an execution time error. You could create a shell script an run that via your command line.
for example (pseudo):
<?php
SELECT * FROM table
LIMIT $x to $y
FOR every result
do some magic, SAVE to record with ID $id
If you call this file my_update.php, run it by typing php path/to/my_update.php and watch the magic. (where php is the php executable).
Be smart and log every action! So when the script fails you have a nice trail and don't have to start all over again! That is exactly why I added the LIMIT in the query so it doesn't have to buffer 11M of rows, but just a few. After the first bunch of rows it will simply go to the next LIMIT. Sort of like pagination, but without visual output.
Sources:
http://php.net/manual/en/features.commandline.php
you need to change the maximum execution time of script from php.ini file.