I have recently switched my database tables from MYISAM to INNODB and experience bad timeouts with queries, mostly inserts. One function I use previously took <2 seconds to insert, delete and update a large collection of records across ~30 MYISAM tables, but now that they are INNODB, the function causes a PHP timeout.
The timeout was set to 60 seconds. I have optimised my script enough that now, even though there are still many queries, they are combined together (multiple inserts, multiple deletes, etc) and the script now takes ~25 seconds, which is a substantial increase from what appeared to be at least 60 seconds.
This duration is still over 10x quicker when previously using MYISAM, is there any mistakes I could be making in the way I process these queries? Or are there any settings that could assist in the performance? Currently the MySQL is using the default settings of installation.
The queries are nothing special, DELETE ... WHERE ... simple logic, same with the INSERT and UPDATE queries.
Hard to say without knowing too much about your environment, but this might be more of a database tuning problem. InnoDB can be VERY slow on budget hardware where every write forces a true flush. (This affects writes, not reads.)
For instance, you may want to read up on options like:
innodb_flush_log_at_trx_commit=2
sync_binlog=0
By avoiding the flushes you may be able to speed up your application considerably, but at the cost of potential data loss if the server crashes.
If data loss is something you absolutely cannot live with, then the other option is to use better hardware.
Run explain for each query. That is, if the slow query is select foo from bar;, run explain select foo from bar;.
Examine the plan, and add indices as necessary. Re-run the explain, and make sure the indices are being used.
Innodb builds hash indexes which helps to speed up lookup by indexes by passing BTREE index and using hash, which is faster
Related
I am running around 50,000 queries on a MyISAM DB however the longer the query runs the slower it becomes. For example, at the beginning 20 rows can be done in a second and by the end it is doing 5 rows a second. Is this happening because MyISAM locks the tables after each insert or is it some other reason?
I think I have to use MyISAM because I am using match()against() with Fulltext indexes, which as far as I know only MyISAM supports. Is it possible to use InnoDB or some other DB type with this functionality or is there a way to speed up the queries as the execution time for the current query is about 45mins.
I anticipate a lot of answers about my query structure, I am collating three tables with different columns into one table and trying to match up the 3 different rows together to form a 'supertable' as a result I need to do a lot of match()against(), preg_match, selects, and looping in order to match them up with accuracy. In order to optimise my loops I am using PDO with prepared statements but this has only brought down the query time to the time I mentioned above, using mysql_connect and the standard functions it was closer to an hour.
Remember the query isn't slow to begin with, only after it's done about 10,000 queries does it slow down noticeably and as it does more it becomes slower and slower until it reaches an unreasonable level for any server company to accept, can anyone give me a solution?
My recommendation is try to update your MySQL,
If you have a lot of writting MySQL -V 5.6+ which allow you to use full text index in InnoDB Tables, beside that Innodb allow you to make rows lock so that is a HUGE thing.
You can also try to repair index with ANALIZE that could help a little bit the performance
MySQL Full text search can't efficiently handle big data by design. Switching from MyISAM to InnoDB is good idea most of the time as it will help with non-FT queries performance and concurrency and will keep you data safe as InnoDB is supporting transaction, but will not boost your FT queries speed much (according to Percona's benchmarks http://bit.ly/M6DMsj ).
I would suggest to move Full-Text queries out of MySQL. Any external search engine like Solr or Sphinx will be very helpful in this case.
I'm not an expert with Solr, but in case you decided to use Sphinx you could use http://astellar.com/2011/12/replacing-mysql-full-text-search-with-sphinx/ as a guide for initial configuration.
Hope this helps.
I have a PHP script that in every run, inserts a new row to a Mysql db (with a relative small amount of data..)
I have more than 20 requests per second, and this is causing my CPU to scream for help..
I'm using the sql INSERT DELAYED method with a MyISAM engine (although I just notice that INSERT DELAYED is not working with MyISAM).
My main concern is my CPU load and I started to look for ways to store this data with more CPU friendly solutions.
My first idea was to write this data to an hourly log files and once an hour to retrieve the data from the logs and insert it to the DB at once.
Maybe a better idea is to use NoSQL DB instead of log files and then once an hour to insert the data from the NoSQL to the Mysql..
I didn't test yet any of these ideas, so I don't really know if this will manage to decrease my CPU load or not. I wanted to ask if someone can help me find the right solution that will have the lowest affect over my CPU.
I recently had a very similar problem and my solution was to simply batch the requests. This sped things up about 50 times because of the reduced overhead of mysql connections and also the greatly decreased amount of reindexing. Storing them to a file then doing one larger (100-300 individual inserts) statement at once probably is a good idea. To speed things up even more turn off indexing for the duration of the insert with
ALTER TABLE tablename DISABLE KEYS
insert statement
ALTER TABLE tablename ENABLE KEYS
doing the batch insert will reduce the number of instances of the php script running, it will reduce the number of currently open mysql handles (large improvement) and it will decrease the amount of indexing.
Ok guys, I manage to lower the CPU load dramatically with APC-cache
I'm doing it like so:
storing the data in memory with APC-cache, with TTL of 70 seconds:
apc_store('prfx_SOME_UNIQUE_STRING', $data, 70);
once a minute I'm looping over all the records in the cache:
$apc_list=apc_cache_info('user');
foreach($apc_list['cache_list'] as $apc){
if((substr($apc['info'],0,5)=='prfx_') && ($val=apc_fetch($apc['info']))){
$values[]=$val;
apc_delete($apc['info']);
}
}
inserting the $values to the DB
and the CPU continues to smile..
enjoy
I would insert a sleep(1); function at the top of your PHP script, before every insert at the top of your loop where 1 = 1 second. This only allows the loop to cycle once per second.
This way it will regulate a bit just how much load the CPU is getting, this would be ideal assuming your only writing a small number of records in each run.
You can read more about the sleep function here : http://php.net/manual/en/function.sleep.php
It's hard to tell without profiling both methods, if you write to a log file first you could end up just making it worse as your turning your operation count from N to N*2. You gain a slight edge by writing it all to a file and doing a batch insert but bear in mind that as the log file fills up it's load/write time increases.
To reduce database load, look at using mem cache for database reads if your not already.
All in all though your probably best of just trying both and seeing what's faster.
Since you are trying INSERT DELAYED, I assume you don't need up to the second data. If you want to stick with MySQL, you can try using replication and the BLACKHOLE table type. By declaring a table as type BLACKHOLE on one server, then replicating it to a MyISAM or other table type on another server, you can smooth out CPU and io spikes. BLACKHOLE is really just a replication log file, so "inserts" into it are very fast and light on the system.
I do not know what is your table size or your server capabilities but I guess you need to make a lot of inserts per single table. In such a situation I would recommend checking for the construction of vertical partitions that will reduce the physical size of each partition and significantly reduce the insertion time to the table.
I currently have 2000 records in a postgresql database being updated every minute that are filtered with a SQL statement. Upto 1000 different filter combinations can exist and approx 500 different filters can be called every minute. At the moment http responses are cached for 59 seconds to ease server load and database calls. However im considering caching the whole db table in memcached and doing the filtering in php. 2000 rows isnt alot but the response time for getting data from memory vs the db would be alot faster.
Would the php processing time outweigh the database response time for sql filtering for this number of rows? The table shouldnt grow anymore than 3000 rows in the foreseeable future.
As with any question relating to is x faster than y, the only real answer is to benchmark it for yourself. However, if the database is properly indexed for the queries you need to perform, it is likely to be quite a bit faster at filtering result sets than most any PHP code you could write.
The RDBMS is on the other hand, is already designed and optimized for locating, filtering, and ordering rows.
The way PostgreSQL operates, if you aren't extremely starving it for memory, 100% of such a small and frequently queried table will be held in RAM (Cache) already by the default caching algorithms. Having the database engine filter it is almost certainly faster than doing the same it in your application.
You may want to inspect your postgresql.conf, especially shared_buffers, the planner cost constants (set random_page_cost almost or exactly as low as seq_page_cost) and effective_cache_size (set it high enough).
You could probably benefit from optimizing indexes. There is a wide range of types available. Consider partial indexes, indexes on expression or multi-column indexes in addition to plain indexes. Test with EXPLAIN ANALYZE and only keep indexes that actually get used and speed up queries. As all of the table resides in RAM, the query planner should calculate that random access is almost or exactly as fast as sequential access. The difference only applies to disc reads.
As you updating every minute, be sure not to keep any indexes that aren't actually helping. Also, vacuuming and analyzing it frequently are keys to performance in such a case. Not VACUUM FULL ANALYZE, just VACUUM ANALYZE. Or use auto-vacuum with tuned settings.
Of course, all the standard advice on performance optimization applies.
I have a query which takes a very long time to run but produces a new table in the end. The actual joins are not all that slow but it spends almost all of its time in 'copying to tmp table' and during this time the status of all other queries (which should be going to unrelated tables) is 'locked'. I am in the process of optimizing the long query but it is ok for it to take a while since it is an offline process, but it is NOT ok for it to stop all other queries which should not be related to it anyway. Does anyone know why all other unrelated queries would comeback as 'locked' and how to prevent this behavior?
You are right in that "unrelated tables" shouldn't be affected. They shouldn't and to my knowledge they aren't.
There is a lot of information over at MySQL regarding locks, storage engines and ways of dealing with it.
To limit locks I would suggest that you write an application that reads all data needed to do this new table and simply have your application insert values to the new table. This might take longer but it will do it in smaller chunks and have less or no locks.
Good luck!
What is your MySQL Version?
Do you use MyISAM? MyISAM has a big LOCK problems on large SELECT commands.
Do you have a dedicated server? what is your maximum size for in-memory tables (look in my.cnf)?
Right guys,
I have a MySQL database, using InnoDB on tables, every so often I have to perform a big cron job that does a large batch of queries and inserts. When I run this cron job, for the 5minutes or so that it is running, no other page is able to load. As soon as it is done, the queries are executed and the page loads.
The table that is actually having all this data added to it, isn't even being queried by the main site. It simply is that when MySQL is under a lot of work, the rest of the site is untouchable. This surely must not be right, what could be causing this to happen? CPU usage for MySQLD rockets to huge figures like 120% (!!!!!) and all MySQL queries are locked.
What could cause/fix this?
No, that's obviously wrong. This is probably related to bad configuration. Take a look at the size of the innodb buffer pool and see if it can be increased. This sounds like a typical case of ram shortage. Healthy setups are almost never cpu bound, and certainly not when doing bulk inserts.
With InnoDB, other things should still be able to access the database. Are you prepared to show the schema (or relevant part of it) and the relevant parts of the application?
Maybe it's contention in hardware.
How big are the transactions which your "cron" job is using? Using tiny transactions will create a massive amount of IO needlessly.
Do your database servers have battery backed raid controllers (assuming your servers use hard drives not SSD)? If not, commits will be quite slow.
How much ram is in your database server? If possible, ensure that it is a bit bigger than your database and set innodb_buffer_pool to > data size - this will mean that read workloads come out of ram anyway, which should make them fast.
Can you reproduce the problem in a test system on production-grade hardware?
I think you might need to re-think how you are building up your queries. InnoDB has page-level locks, but with massive updates you can still lock down quite a bit of your queries.
Post your actual queries, and try again.. I don't think there is a generic solution for a generic question like this, so look into optimizing what you're doing today.
You could have the script delay for 1/10 second or so between each query. It will take longer but allow activity in the background.
sleep( 0.1 );
You will probably only need to do this for the writes, reads are very cheap.