So we have a big query that takes up to 3 mins to complete, and multiple tiny ones after.
Initially, we were having timeouts on the first query, so increasing timeout variables in my.cnf + changing some settings in our firewall (php and mysql are in 2 different CentOS7 VMs, and they connect with pfSense - HAProxy) did the trick.
However now, we get "mysql has gone away" on small queries following anything that takes above 30 secs, as if Mysql needed a break. (We're keeping the same connection).
I did a simple test like the following:
SELECT SLEEP(180), 123 AS `abc` <-- Used to yield gone away WITHOUT outputing "0|123"
SELECT SLEEP(1), 456 AS `abc`
It used to yield gone aways, and now it's working just fine
Now for the 2 other scenarios, this one works:
SELECT SLEEP(29), 123 AS `abc` <-- Ouputs "0|123"
SELECT SLEEP(5), 456 AS `abc` <-- Outputs "0|456"
but this one dies on the 2nd statement:
SELECT SLEEP(31), 123 AS `abc` <-- Ouputs "0|123"
SELECT SLEEP(1), 456 AS `abc` <-- Gone away
Part of what I put in my.cnf that I think is relevant:
wait_timeout=1800
net_read_timeout=1800
net_write_timeout=1800
slave_net_timeout=1800
max_connections=10000
innodb_log_file_size=512M
innodb_buffer_pool_size=1G
innodb_lock_wait_timeout=1800
max_allowed_packet=256M
group_concat_max_len=22000000
lock_wait_timeout=3153600
And the full output (from PHPMyAdmin):
alter algorithm DEFAULT
aria block size 8,192
aria checkpoint interval 30
aria checkpoint log activity 1,048,576
aria encrypt tables OFF
aria force start after recovery failures 0
aria group commit none
aria group commit interval 0
aria log file size 1,073,741,824
aria log purge type immediate
aria max sort file size 9,223,372,036,853,727,232
aria page checksum ON
aria pagecache age threshold 300
aria pagecache buffer size 134,217,728
aria pagecache division limit 100
aria pagecache file hash size 512
aria recover options BACKUP,QUICK
aria repair threads 1
aria sort buffer size 268,434,432
aria stats method nulls_unequal
aria sync log dir NEWFILE
aria used for temp tables ON
auto increment increment 1
auto increment offset 1
autocommit ON
automatic sp privileges ON
back log 900
basedir /usr/
big tables OFF
bind address 10.2.0.154
binlog annotate row events ON
binlog cache size 32,768
binlog checksum CRC32
binlog commit wait count 0
binlog commit wait usec 100,000
binlog direct non transactional updates OFF
binlog file cache size 16,384
binlog format MIXED
binlog optimize thread scheduling ON
binlog row image FULL
binlog stmt cache size 32,768
bulk insert buffer size 8,388,608
character set client latin1
character set client (Session value) utf8mb4
character set connection latin1
character set connection (Session value) utf8mb4
character set database latin1
character set filesystem binary
character set results latin1
character set results (Session value) utf8mb4
character set server latin1
character set system utf8
character sets dir /usr/share/mysql/charsets/
check constraint checks ON
collation connection latin1_swedish_ci
collation connection (Session value) utf8mb4_unicode_ci
collation database latin1_swedish_ci
collation server latin1_swedish_ci
column compression threshold 100
column compression zlib level 6
column compression zlib strategy DEFAULT_STRATEGY
column compression zlib wrap OFF
completion type NO_CHAIN
concurrent insert AUTO
connect timeout 10
core file OFF
datadir /var/lib/mysql/
date format %Y-%m-%d
datetime format %Y-%m-%d %H:%i:%s
deadlock search depth long 15
deadlock search depth short 4
deadlock timeout long 50,000,000
deadlock timeout short 10,000
debug no thread alarm OFF
default regex flags
default storage engine InnoDB
default tmp storage engine
default week format 0
delay key write ON
delayed insert limit 100
delayed insert timeout 300
delayed queue size 1,000
div precision increment 4
encrypt binlog OFF
encrypt tmp disk tables OFF
encrypt tmp files OFF
enforce storage engine
eq range index dive limit 0
event scheduler OFF
expensive subquery limit 100
expire logs days 0
explicit defaults for timestamp OFF
extra max connections 1
extra port 0
flush OFF
flush time 0
foreign key checks ON
ft boolean syntax + -><()~*:""&|
ft max word len 84
ft min word len 4
ft query expansion limit 20
ft stopword file (built-in)
general log OFF
general log file /var/lib/mysql/_logs/general
group concat max len 22,000,000
gtid binlog pos
gtid binlog state
gtid current pos 0-2-77303
gtid domain id 0
gtid ignore duplicates OFF
gtid pos auto engines
gtid slave pos 0-2-77303
gtid strict mode OFF
have compress YES
have crypt YES
have dynamic loading YES
have geometry YES
have openssl YES
have profiling YES
have query cache YES
have rtree keys YES
have ssl DISABLED
have symlink YES
histogram size 0
histogram type SINGLE_PREC_HB
host cache size 1,103
hostname kb-malga-sql1
idle readonly transaction timeout 0
idle transaction timeout 0
idle write transaction timeout 0
ignore builtin innodb OFF
ignore db dirs
in predicate conversion threshold 1,000
init connect
init file
init slave
innodb adaptive flushing ON
innodb adaptive flushing lwm 10
innodb adaptive hash index ON
innodb adaptive hash index parts 8
innodb adaptive max sleep delay 150,000
innodb autoextend increment 64
innodb autoinc lock mode 1
innodb background scrub data check interval 3,600
innodb background scrub data compressed OFF
innodb background scrub data interval 604,800
innodb background scrub data uncompressed OFF
innodb buf dump status frequency 0
innodb buffer pool chunk size 134,217,728
innodb buffer pool dump at shutdown ON
innodb buffer pool dump now OFF
innodb buffer pool dump pct 25
innodb buffer pool filename ib_buffer_pool
innodb buffer pool instances 8
innodb buffer pool load abort OFF
innodb buffer pool load at startup ON
innodb buffer pool load now OFF
innodb buffer pool size 1,073,741,824
innodb change buffer max size 25
innodb change buffering all
innodb checksum algorithm crc32
innodb checksums ON
innodb cmp per index enabled OFF
innodb commit concurrency 0
innodb compression algorithm zlib
innodb compression default OFF
innodb compression failure threshold pct 5
innodb compression level 6
innodb compression pad pct max 50
innodb concurrency tickets 5,000
innodb data file path ibdata1:12M:autoextend
innodb data home dir
innodb deadlock detect ON
innodb default encryption key id 1
innodb default row format dynamic
innodb defragment OFF
innodb defragment fill factor 1
innodb defragment fill factor n recs 20
innodb defragment frequency 40
innodb defragment n pages 7
innodb defragment stats accuracy 0
innodb disable sort file cache OFF
innodb disallow writes OFF
innodb doublewrite ON
innodb encrypt log OFF
innodb encrypt tables OFF
innodb encrypt temporary tables OFF
innodb encryption rotate key age 1
innodb encryption rotation iops 100
innodb encryption threads 0
innodb fast shutdown 1
innodb fatal semaphore wait threshold 600
innodb file format
innodb file per table ON
innodb fill factor 100
innodb flush log at timeout 1
innodb flush log at trx commit 1
innodb flush method fsync
innodb flush neighbors 1
innodb flush sync ON
innodb flushing avg loops 30
innodb force load corrupted OFF
innodb force primary key OFF
innodb force recovery 0
innodb ft aux table
innodb ft cache size 8,000,000
innodb ft enable diag print OFF
innodb ft enable stopword ON
innodb ft max token size 84
innodb ft min token size 3
innodb ft num word optimize 2,000
innodb ft result cache limit 2,000,000,000
innodb ft server stopword table
innodb ft sort pll degree 2
innodb ft total cache size 640,000,000
innodb ft user stopword table
innodb idle flush pct 100
innodb immediate scrub data uncompressed OFF
innodb instant alter column allowed add_last
innodb io capacity 200
innodb io capacity max 2,000
innodb large prefix
innodb lock schedule algorithm fcfs
innodb lock wait timeout 1,800
innodb locks unsafe for binlog OFF
innodb log buffer size 16,777,216
innodb log checksums ON
innodb log compressed pages ON
innodb log file size 536,870,912
innodb log files in group 2
innodb log group home dir ./
innodb log optimize ddl OFF
innodb log write ahead size 8,192
innodb lru scan depth 1,024
innodb max dirty pages pct 75
innodb max dirty pages pct lwm 0
innodb max purge lag 0
innodb max purge lag delay 0
innodb max purge lag wait 4,294,967,295
innodb max undo log size 10,485,760
innodb monitor disable
innodb monitor enable
innodb monitor reset
innodb monitor reset all
innodb old blocks pct 37
innodb old blocks time 1,000
innodb online alter log max size 134,217,728
innodb open files 2,000
innodb optimize fulltext only OFF
innodb page cleaners 4
innodb page size 16,384
innodb prefix index cluster optimization OFF
innodb print all deadlocks OFF
innodb purge batch size 300
innodb purge rseg truncate frequency 128
innodb purge threads 4
innodb random read ahead OFF
innodb read ahead threshold 56
innodb read io threads 4
innodb read only OFF
innodb replication delay 0
innodb rollback on timeout OFF
innodb rollback segments 128
innodb scrub log OFF
innodb scrub log speed 256
innodb sort buffer size 1,048,576
innodb spin wait delay 4
innodb stats auto recalc ON
innodb stats include delete marked OFF
innodb stats method nulls_equal
innodb stats modified counter 0
innodb stats on metadata OFF
innodb stats persistent ON
innodb stats persistent sample pages 20
innodb stats sample pages 8
innodb stats traditional ON
innodb stats transient sample pages 8
innodb status output OFF
innodb status output locks OFF
innodb strict mode OFF
innodb sync array size 1
innodb sync spin loops 30
innodb table locks ON
innodb temp data file path ibtmp1:12M:autoextend
innodb thread concurrency 0
innodb thread sleep delay 10,000
innodb tmpdir
innodb undo directory ./
innodb undo log truncate OFF
innodb undo logs 128
innodb undo tablespaces 0
innodb use atomic writes ON
innodb use native aio ON
innodb version 10.3.29
innodb write io threads 4
interactive timeout 28,800
join buffer size 262,144
join buffer space limit 2,097,152
join cache level 2
keep files on create OFF
key buffer size 134,217,728
key cache age threshold 300
key cache block size 1,024
key cache division limit 100
key cache file hash size 512
key cache segments 0
large files support ON
large page size 0
large pages OFF
lc messages en_US
lc messages dir
lc time names en_US
license GPL
local infile ON
lock wait timeout 3,153,600
locked in memory OFF
log bin OFF
log bin basename
log bin compress OFF
log bin compress min len 256
log bin index
log bin trust function creators OFF
log disabled statements sp
log error /var/lib/mysql/_logs/error.err
log output FILE
log queries not using indexes ON
log slave updates OFF
log slow admin statements ON
log slow disabled statements sp
log slow filter admin,filesort,filesort_on_disk,filesort_priority_queue,full_join,full_scan,not_using_index,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk
log slow rate limit 5
log slow slave statements ON
log slow verbosity query_plan,explain
log tc size 24,576
log warnings 2
long query time 5
low priority updates OFF
lower case file system OFF
lower case table names 1
master verify checksum OFF
max allowed packet 268,435,456
max binlog cache size 18,446,744,073,709,547,520
max binlog size 1,073,741,824
max binlog stmt cache size 18,446,744,073,709,547,520
max connect errors 100
max connections 10,000
max delayed threads 20
max digest length 1,024
max error count 64
max heap table size 16,777,216
max insert delayed threads 20
max join size 18,446,744,073,709,551,616
max length for sort data 1,024
max long data size 268,435,456
max prepared stmt count 16,382
max recursive iterations 4,294,967,295
max relay log size 1,073,741,824
max seeks for key 4,294,967,295
max session mem used 9,223,372,036,854,775,808
max sort length 1,024
max sp recursion depth 0
max statement time 0
max tmp tables 32
max user connections 0
max write lock count 4,294,967,295
metadata locks cache size 1,024
metadata locks hash instances 8
min examined row limit 0
mrr buffer size 262,144
multi range count 256
myisam block size 1,024
myisam data pointer size 6
myisam max sort file size 9,223,372,036,853,727,232
myisam mmap size 18,446,744,073,709,551,616
myisam recover options BACKUP,QUICK
myisam repair threads 1
myisam sort buffer size 134,216,704
myisam stats method NULLS_UNEQUAL
myisam use mmap OFF
mysql56 temporal format ON
net buffer length 16,384
net read timeout 1,800
net retry count 10
net write timeout 1,800
old OFF
old alter table DEFAULT
old mode
old passwords OFF
open files limit 32,768
optimizer prune level 1
optimizer search depth 62
optimizer selectivity sampling limit 100
optimizer switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
optimizer use condition selectivity 1
performance schema OFF
performance schema accounts size -1
performance schema digests size -1
performance schema events stages history long size -1
performance schema events stages history size -1
performance schema events statements history long size -1
performance schema events statements history size -1
performance schema events waits history long size -1
performance schema events waits history size -1
performance schema hosts size -1
performance schema max cond classes 80
performance schema max cond instances -1
performance schema max digest length 1,024
performance schema max file classes 50
performance schema max file handles 32,768
performance schema max file instances -1
performance schema max mutex classes 200
performance schema max mutex instances -1
performance schema max rwlock classes 40
performance schema max rwlock instances -1
performance schema max socket classes 10
performance schema max socket instances -1
performance schema max stage classes 160
performance schema max statement classes 200
performance schema max table handles -1
performance schema max table instances -1
performance schema max thread classes 50
performance schema max thread instances -1
performance schema session connect attrs size -1
performance schema setup actors size 100
performance schema setup objects size 100
performance schema users size -1
pid file /var/lib/mysql/kb-malga-sql1.pid
plugin dir /usr/lib64/mysql/plugin/
plugin maturity gamma
port 3,306
preload buffer size 32,768
profiling OFF
profiling history size 15
progress report time 5
protocol version 10
proxy protocol networks
query alloc block size 16,384
query cache limit 1,048,576
query cache min res unit 4,096
query cache size 1,048,576
query cache strip comments OFF
query cache type OFF
query cache wlock invalidate OFF
query prealloc size 24,576
range alloc block size 4,096
read binlog speed limit 0
read buffer size 131,072
read only OFF
read rnd buffer size 262,144
relay log
relay log basename
relay log index
relay log info file relay-log.info
relay log purge ON
relay log recovery OFF
relay log space limit 0
replicate annotate row events ON
replicate do db
replicate do table
replicate events marked for skip REPLICATE
replicate ignore db
replicate ignore table
replicate wild do table
replicate wild ignore table
report host
report password
report port 3,306
report user
rowid merge buff size 8,388,608
rpl semi sync master enabled OFF
rpl semi sync master timeout 10,000
rpl semi sync master trace level 32
rpl semi sync master wait no slave ON
rpl semi sync master wait point AFTER_COMMIT
rpl semi sync slave delay master OFF
rpl semi sync slave enabled OFF
rpl semi sync slave kill conn timeout 5
rpl semi sync slave trace level 32
secure auth ON
secure file priv
secure timestamp NO
server id 1
session track schema ON
session track state change OFF
session track system variables autocommit,character_set_client,character_set_connection,character_set_results,time_zone
session track transaction info OFF
skip external locking ON
skip name resolve OFF
skip networking OFF
skip show database OFF
slave compressed protocol OFF
slave ddl exec mode IDEMPOTENT
slave domain parallel threads 0
slave exec mode STRICT
slave load tmpdir /tmp
slave max allowed packet 1,073,741,824
slave net timeout 1,800
slave parallel max queued 131,072
slave parallel mode conservative
slave parallel threads 0
slave parallel workers 0
slave run triggers for rbr NO
slave skip errors OFF
slave sql verify checksum ON
slave transaction retries 10
slave transaction retry errors 1213,1205
slave transaction retry interval 0
slave type conversions
slow launch time 2
slow query log OFF
slow query log file /var/lib/mysql/_logs/slow
socket /var/lib/mysql/mysql.sock
sort buffer size 2,097,152
sql auto is null OFF
sql big selects ON
sql buffer result OFF
sql log bin ON
sql log off OFF
sql mode ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
sql notes ON
sql quote show create ON
sql safe updates OFF
sql select limit 18,446,744,073,709,551,616
sql slave skip counter 0
sql warnings OFF
ssl ca
ssl capath
ssl cert
ssl cipher
ssl crl
ssl crlpath
ssl key
standard compliant cte ON
storage engine InnoDB
stored program cache 256
strict password validation ON
sync binlog 0
sync frm ON
sync master info 10,000
sync relay log 10,000
sync relay log info 10,000
system time zone EDT
system versioning alter history ERROR
system versioning asof DEFAULT
table definition cache 400
table open cache 2,000
table open cache instances 5
tcp keepalive interval 0
tcp keepalive probes 0
tcp keepalive time 0
thread cache size 256
thread concurrency 10
thread handling one-thread-per-connection
thread pool idle timeout 60
thread pool max threads 65,536
thread pool oversubscribe 3
thread pool prio kickup timer 1,000
thread pool priority auto
thread pool size 2
thread pool stall limit 500
thread stack 299,008
time format %H:%i:%s
time zone SYSTEM
timed mutexes OFF
tmp disk table size 18,446,744,073,709,551,616
tmp memory table size 16,777,216
tmp table size 16,777,216
tmpdir /tmp
transaction alloc block size 8,192
transaction prealloc size 4,096
tx isolation REPEATABLE-READ
tx read only OFF
unique checks ON
updatable views with limit YES
use stat tables NEVER
userstat OFF
version 10.3.29-MariaDB
version comment MariaDB Server
version compile machine x86_64
version compile os Linux
version malloc library system
version source revision 4f143a88bcb36e94e9edba8a3c5b4a350dcd9bf9
version ssl library OpenSSL 1.0.2k-fips 26 Jan 2017
wait timeout 1,800
wsrep osu method TOI
wsrep auto increment control ON
wsrep causal reads OFF
wsrep certification rules strict
wsrep certify nonpk ON
wsrep cluster address
wsrep cluster name my_wsrep_cluster
wsrep convert lock to trx OFF
wsrep data home dir /var/lib/mysql/
wsrep dbug option
wsrep debug OFF
wsrep desync OFF
wsrep dirty reads OFF
wsrep drupal 282555 workaround OFF
wsrep forced binlog format NONE
wsrep gtid domain id 0
wsrep gtid mode OFF
wsrep load data splitting ON
wsrep log conflicts OFF
wsrep max ws rows 0
wsrep max ws size 2,147,483,647
wsrep mysql replication bundle 0
wsrep node address
wsrep node incoming address AUTO
wsrep node name kb-malga-sql1
wsrep notify cmd
wsrep on OFF
wsrep patch version wsrep_25.24
wsrep provider none
wsrep provider options
wsrep recover OFF
wsrep reject queries NONE
wsrep replicate myisam OFF
wsrep restart slave OFF
wsrep retry autocommit 1
wsrep slave fk checks ON
wsrep slave uk checks OFF
wsrep slave threads 1
wsrep sst auth
wsrep sst donor
wsrep sst donor rejects queries OFF
wsrep sst method rsync
wsrep sst receive address AUTO
wsrep start position 00000000-0000-0000-0000-000000000000:-1
wsrep sync wait 0
EDIT
In php.ini:
max_execution_time: 1600
max_input_time: 1600
memory_limit: 4096M
session.gc_maxlifetime: 10800
My software makes a lot of MySQL queries to my server, and I have never had any issues in the past with it, but just recently nothing was loading, no webpages, no SQL was running, nothing. I managed to get on WHM for my server and kill the process, only to watch it spike back up to 300%. Nothing I have been able to do has made it go down. What information do I need to share to get help with this? I am not a sys admin nor do I have one or resources for one. I wouldn't usually be asking for help and just optimize all my queries for something like this as it wasn't a problem for the past 3 months but suddenly became one out of nowhere, at least not that I noticed. At this point my program is saying that one of my database tables has crashed and needs repaired... What can I do? Thanks in advance for any help...
I have already considered optimization but I was hoping for a quick solution to implement as I have customers waiting, then I can spend a few days working on optimizing my SQL that, like I said, wasn't having any issues before. I am confused about it.
Also I am not sure if this helps but tracing the process in WHM prints this repeatedly and nothing else:
fcntl(16, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(16, F_SETFL, O_RDWR|O_NONBLOCK) = 0
accept(16, {sa_family=AF_LOCAL, NULL}, [2]) = 35
fcntl(16, F_SETFL, O_RDWR) = 0
setsockopt(35, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported)
futex(0x13298a4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x13298a0, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x1327240, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=14, events=POLLIN}, {fd=16, events=POLLIN}], 2, -1) = 1 ([{fd=16, revents=POLLIN}])
/etc/my.conf
innodb_file_per_table=1
default-storage-engine=MyISAM
performance-schema=0
max_allowed_packet=268435456
open_files_limit=10000
This is all that is available to me as far as my.conf file. The error log doesn't exist in /var/log so I don't have anything to give in that regard...
SQL version:
[Server] # mysql -V
mysql Ver 14.14 Distrib 5.6.41, for Linux (x86_64) using EditLine wrapper
I have an additional question or add-on to this. I don't know if it makes much of a difference but, say my code is running using 30% CPU on the mysql process, I can actually turn off the code and the mysql process CPU usage will not change. What does this mean?
Edit: (these are all expiring within a week from 12/09/2018)
Global Status
Current Settings
ulimit -a
df -h
mysqltuner report
The my.cnf file contents that I listed is all that was there. Nothing else. I will get the top command and iostat -xm 5 3 when I am running the software full speed again to see the results.
Rate Per Second=RPS Suggestions to consider based on your Linux ulimit -a report.
ulimit -n 16384 to raise Open Files limit from 1024 to support your activities.
For this to persist over Linux Shutdown/Restart, review this url.
https://glassonionblog.wordpress.com/2013/01/27/increase-ulimit-and-file-descriptors-limit/
Your specifics may be slightly different due to version of Linux.
Suggestions to consider for your my.cnf [mysqld] section
innodb_lru_scan_depth=100 # from 1024 to reduce CPU busy every second. 93% savings for this one function.
thread_cache_size=32 # from 9 for thread breathing room and growth.
innodb_io_capacity=1800 # from 200 to take advantage of your HDD IOPS capacity
key_cache_age_threshold=7200 # from 300 seconds to reduce key_reads RPS of 16
query_cache_size=0 # from 1M to conserve RAM - QC is OFF and not used
query_cache_limit=0 # from 1M to conserve RAM - QC is OFF and not used
key_buffer_size=128M # from 8M which had NO free space at the end of your work day
For additional suggestions, see my profile, Network profile for contact information.
Lately my site (with 260000 posts, 12000 images, 2,360,987 mysql rows and 450.7 MiB size) is running slow and at times not loading for many mins
I installed this Debug bar plugin https://wordpress.org/plugins/debug-bar/
Memory usage
on server is: 174,319,288 bytes
Intel(R) Xeon(R) CPU E3-1230 V2 # 3.30GHz , 16 GB
(PHP: 5.5.23, MySQL: 5.6.23, Apache 2.4)
Even tried disabling all plugins it doesnt help much... it comes down 160-163,xxx,xxx bytes
on wamp is : 37,834,920 bytes
(PHP: 5.5.12, MySQL: 5.6.17)
Why the difference is huge? How to detect the problem?
Been using the following plugins
Acunetix WP Security
Akismet
Antispam Bee
CloudFlare
Contact Form 7
Custom Post Type UI
Debug Bar
Login LockDown
Redirection
Theme Test Drive
W3 Total Cache
WordPress SEO
WP-Optimize
WP Missed Schedule
my.cnf values for the above server are
[mysqld]
slow-query-log=1
long-query-time=1
slow-query-log-file="/var/log/mysql-slow.log"
default-storage-engine = MyISAM
local-infile = 0
innodb_buffer_pool_size = 1G
innodb_log_file_size = 256M
innodb_file_per_table=1
innodb_stats_on_metadata=0
max_connections=360
wait_timeout=60
connect_timeout = 15
thread_cache_size=20
thread_concurrency=8
key_buffer_size = 1024M
join_buffer_size = 2M
sort_buffer_size=1M
query_cache_limit=64M
query_cache_size=128M
query_cache_type=1
max_heap_table_size=32M
tmp_table_size=32MB
table_open_cache=1000
table_definition_cache=1024
open_files_limit=10000
max_allowed_packet=268435456
low_priority_updates=1
concurrent_insert=2
#port = 8881
#innodb_force_recovery=0
#innodb_purge_threads=0
The "server" has Apache; that accounts for some (all?) of the difference.
Windows and Unix handle memory differently, and measure it differently. So, the difference may be irrelevant.
The numbers you have quoted are not big; what is the problem?
"Tried restarting the server and checked it in initial moments" -- That's mostly irrelevant. Programs tend to grow over time, up to some limit. Let's see the values in "steady state" with a typical load.
You have enough RAM to cache the entire dataset in RAM. But, due to inactivity, probably most of the data is not touched, hence has not been read into cache.
"High" memory usage is when you are swapping. Actually, that is probably "too high". So, say, 90% is "high". Your numbers are nowhere near that.
innodb_buffer_pool_size=200M -- is not enough to hold the entire 450.7MB dataset, but, as I say, most of the data is probably not actively used.
Edit (after posting of settings)
table_cache=10M
That is terrible! You won't be opening 10 million tables. Set it to 1000.
max_heap_table_size=512M
tmp_table_size=512MB
Those are dangerous. If you have multiple connections, each needing a tmp table (because of a complex query), you could run out of memory fast. Set them down to 32M.
innodb_force_recovery=3
Comment out that line -- It is to be used once, then removed.
The rest of the settings look harmless for this discussion.
I have problems to optimize my VPS MySQL to use. I have a plan in RamNode with the following specs:
- Intel® Xeon® CPU E3-1240 V2 # 3.40GHz (4 Cores)
- 4GB de Ram
- 135 GB SSD Raid 10
I have problems in one of the applications that I have hosted, slow, sometimes gives up error "max user connections".
Below the test conducted in MySQLTunner :
Storage Engine Statistics
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[--] Data in MyISAM tables: 136M (Tables: 300)
[--] Data in InnoDB tables: 44M (Tables: 202)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[!!] Total fragmented tables: 220
Performance Metrics
[--] Up for: 1d 20h 25m 13s (3M q [23.681 qps], 251K conn, TX: 9B, RX: 605M)
[--] Reads / Writes: 57% / 43%
[--] Total buffers: 528.0M global + 3.6M per thread (400 max threads)
[!!] Query cache prunes per day: 7322
[!!] Sorts requiring temporary tables: 69% (144K temp sorts / 208K sorts)
[!!] Joins performed without indexes: 21719
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance. Enable the slow query log to troubleshoot bad queries. Adjust your join queries to always utilize indexes
Variables to adjust:
query_cache_size (> 64M)
sort_buffer_size (> 2M)
read_rnd_buffer_size (> 236K)
join_buffer_size (> 128.0K, or always use indexes with joins)
Below My.CNF
[mysqld]
max_connections = 400
max_user_connections=40
key_buffer_size = 256M
myisam_sort_buffer_size = 16M
read_buffer_size = 1M
table_open_cache = 2048
thread_cache_size = 128
wait_timeout = 20
connect_timeout = 10
tmp_table_size = 128M
max_heap_table_size = 64M
max_allowed_packet=268435456
net_buffer_length = 5500
max_connect_errors = 10
concurrent_insert = 2
read_rnd_buffer_size = 242144
bulk_insert_buffer_size = 2M
query_cache_limit = 2M
query_cache_size = 64M
query_cache_type = 1
query_prealloc_size = 87382
query_alloc_block_size = 21845
transaction_alloc_block_size = 2730
transaction_prealloc_size = 1364
max_write_lock_count = 2
log-error
external-locking=FALSE
open_files_limit=15000
default-storage-engine=MyISAM
innodb_file_per_table=1
[mysqld_safe]
[mysqldump]
quick
max_allowed_packet = 8M
[isamchk]
key_buffer = 128M
sort_buffer = 128M
read_buffer = 64M
write_buffer = 64M
[myisamchk]
key_buffer = 128M
sort_buffer = 128M
read_buffer = 64M
write_buffer = 64M
#### Per connection configuration ####
sort_buffer_size = 2M
join_buffer_size = 2M
thread_stack = 192K
log-slow-queries
If you can help me thank you :)
You seem to be using a mixture of MyISAM and InnoDB tables. It would be best to settle on one or the other - almost certainly InnoDB is going to be better for optimisation purposes.
You should convert all tables in your solution to InnoDB. If your solution can create new tables, change the default-storage-engine type to InnoDB as well.
default-storage-engine=InnoDB
Then, since you have plenty of RAM, tune to ensure that all your data fits in RAM, so MySQL doesn't have to keep doing expensive disk reads all the time. Set innodb_buffer_pool_size to be at least 1G, maybe more if your database is going to grow rapidly. You can probably safely go up to 2G, so long as you don't have anything else that's really ram intensive running on this VPS.
innodb_buffer_pool_size=2G
Currently you are running with the default InnoDB config options, which looks OK (144MB pool size, data size of only 44Mb), but you may as well configure for growth and take advantage of the RAM you have at your disposal. Also if you convert the 140Mb of MyISAM tables to InnoDB (recommended), then you really need this figure to be higher. This setting is probably the most important, and likely to make the biggest difference to performance.
Also, as you are hitting the max number of connections, you need to increase that value too. Your VPS should be able to handle more than the default of 40 - try anywhere between 50-100, depending on how many concurrent users/connections you expect.
max_user_connections=100
The stats also show that you have a large number of queries being executed without indexes
[!!] Joins performed without indexes: 21719
This needs attention. If it's not obvious to work out which fields need indexing (normally any fields used in join statements, and some fields used for searching and filtering, especially if they are numeric and have only a limited selection of values), you can try running queries in your favourite mysql client, with the EXPLAIN statement. This will give you detailed information about which parts of the query are performing poorly.
See the Mysql manual https://dev.mysql.com/doc/refman/5.6/en/explain.html for more information.
It is always worth trying the addition of indexes to see whether they improve query performance. If they don't, remove them again, as indexes can make insert/update operations more expensive in terms of server resource (as the indexes need to be updated as well as the underlying data).
I highly recommend a MySQL GUI such as sqlyog for playing around with indexing. The MySQL client tools are also OK.
For more details see posts such as https://www.percona.com/blog/2007/11/01/innodb-performance-optimization-basics/ (a bit old but still talks sense).
If you have a compelling reason why you'd want to use MyISAM in preference to InnoDB (and I really can't think of one), then you would need different advice as the recommendations for innodb pool size are of no use to you.
Run OPTIMIZE TABLE to defragment tables for better performance.
That's bogus advice. It is almost never useful, especially for InnoDB.
2G for the buffer pool is dangerously large for a tiny 4GB system, especially while you have MyISAM tables in use.
While you have a mixture:
key_buffer_size = 200M
innodb_buffer_pool_size = 500M
After you switch to InnoDB:
key_buffer_size = 30M
innodb_buffer_pool_size = 1000M
Swapping is much worse than lowering a value.
See my tips on converting from MyISAM to InnoDB.
The Query cache is doing a lot of prunes. Every write to a table forces a prune of all entries for that table. In general, the QC should be turned off for production servers. Raising the size above 64M would make things worse.
max_user_connections=40
Is one "user" connecting more than 40 times at once? Or do you have Apache with MaxClients > 40?
Show us a couple of your slow queries, together with SHOW CREATE TABLE; we may be able to speed them up.