Query execution time issue - php

I have some trouble with the execution time of a script. The query takes a lot of time. This is the query:
select avg(price) from voiture where duration<30 AND make="Audi" AND model="A4"
+---+---+---+---+---+---+---+---+---+---+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+---+---+---+---+---+---+---+---+---+---+
| 1 | SIMPLE | voiture | ALL | NULL | NULL | NULL | NULL | 1376949 | Using where |
+---+---+---+---+---+---+---+---+---+---+
select price from voiture where duration<30 AND make="Audi" AND model="A4"
+---+---+---+---+---+---+---+---+---+---+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+---+---+---+---+---+---+---+---+---+---+
| 1 | SIMPLE | voiture | ALL | NULL | NULL | NULL | NULL | 1376949 | Using where |
+---+---+---+---+---+---+---+---+---+---+
This query take around 2 seconds to be executed on the phpMyAdmin interface. I tried to see what the issue was and removing the avg function makes the query lasts around 0.0080 seconds.
I asked myself how long it would take to make the calculate the avg in the php script, but the query with or withouth avg takes around 2 seconds both.
So I decided to take all the values of my table and make the process in the script, so I use this query:
select * from voiture where duration<30 AND make="Audi" AND model="A4"
+---+---+---+---+---+---+---+---+---+---+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+---+---+---+---+---+---+---+---+---+---+
| 1 | SIMPLE | voiture | ALL | NULL | NULL | NULL | NULL | 1376949 | Using where |
+---+---+---+---+---+---+---+---+---+---+
On the phpMyAdmin interface, it takes 0.0112 seconds. But in my php script, it takes 25 seconds !
$timestart2=microtime(true);
$querysec='select * from voiture where duration<30 AND make="Audi" AND model="A4"';
$requestsec=mysql_query($querysec) or die(mysql_error());
$timeend2=microtime(true);
$time2=$timeend2-$timestart2;
$page_load_time2 = number_format($time2, 9);
Here is the table structure:
CREATE TABLE `voiture` (
`carid` bigint(9) NOT NULL,
`serviceid` bigint(9) NOT NULL,
`service` varchar(256) NOT NULL,
`model` varchar(256) DEFAULT NULL,
`gearingType` varchar(256) DEFAULT NULL,
`displacement` int(5) DEFAULT NULL,
`cylinders` int(2) DEFAULT NULL,
`fuel` varchar(32) DEFAULT NULL,
`mileage` int(7) DEFAULT NULL,
`existFlag` tinyint(1) DEFAULT NULL,
`lastUpdate` date DEFAULT NULL,
`version` varchar(256) DEFAULT NULL,
`bodyType` varchar(256) DEFAULT NULL,
`firstRegistration` date DEFAULT NULL,
`powerHp` int(4) DEFAULT NULL,
`powerKw` int(4) DEFAULT NULL,
`vat` varchar(256) DEFAULT NULL,
`price` decimal(12,2) DEFAULT NULL,
`duration` int(3) DEFAULT NULL,
`pageUrl` varchar(256) DEFAULT NULL,
`carImg` varchar(256) DEFAULT NULL,
`color` varchar(256) DEFAULT NULL,
`doors` int(1) DEFAULT NULL,
`seats` int(1) DEFAULT NULL,
`prevOwner` int(1) DEFAULT NULL,
`co2` varchar(256) DEFAULT NULL,
`consumption` varchar(256) DEFAULT NULL,
`gears` int(1) DEFAULT NULL,
`equipment` varchar(1024) NOT NULL,
`make` varchar(256) NOT NULL,
`country` varchar(3) NOT NULL
)
There's an index on carid and serviceid
Why does my query takes so long to be executed ? Is there a way it can be improved ?
Why is the execution time different from phpMyAdmin and my php script ?

On the phpMyAdmin interface, it takes 0.0112 seconds. But in my php
script, it takes 25 seconds!
phpMyAdmin interface adds LIMIT to each query. By default it's LIMIT 30.
To decrease time of your aggregate query you need to create indexes for each condition you use(or one composite index, may be).
So, try to create indexes for your model, make and duration fields.
Also, your table is too denormalized. You can to create pair of table to normalize it for a bit.
Ex: Vendors(id, name), Models(id, name) and modify your voiture to have vendor_id/model_id fields instead of text make/model.
Then your initial query will look like:
select avg(t.price) from voiture t
INNER JOIN Models m ON m.id = t.model_id
INNER JOIN Vendors v ON v.id = t.vendor_id
where t.duration<30 AND v.name="Audi" AND m.name="A4"
It will scan light lookup tables for text matches and operate with your heavy table with indexed ids.

There are many possible solution, first thing you can do is create index at DB level this can also improve your execution time.
second this check server, there may be some processes which would occupying your server resources, and making your server slow.

You can make it faster using GROUP BY like
select AVG(price) from voiture where duration<30 AND make="Audi" AND model="A4" GROUP BY make
Also you can add an index for make column
ALTER TABLE `voiture` ADD INDEX `make` (`make`)

Related

Mysql query running COUNT every time the same value is fetched

I am currently using Laravels QueryBuilder to fetch all users and the number of open tickets they have.
The query looks like this:
return DB::table('users')
->leftjoin('tickets', 'users.organisation', '=', 'tickets.organisation')
->leftjoin('tickets_statuses', 'tickets.ticket_id', '=', 'tickets_statuses.ticket_id')
->select(
'users.organisation',
DB::raw('COUNT(tickets.ticket_id) as tickets')
)
->where('tickets_statuses.status_id', 0)
->where('users.admin', false)
->groupby('users.organisation')
->get();
The issue is, I may have 6 users that belong to the same organisation. So if 80 open tickets belong to the organisation, the query would do 80 x 6 for some reason resulting in 480 open ticket count when it should be 80.
/** EDIT **/
-- ----------------------------
-- Table structure for users
-- ----------------------------
DROP TABLE IF EXISTS `users`;
CREATE TABLE `users` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`email` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`password` varchar(60) COLLATE utf8_unicode_ci NOT NULL,
`organisation` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`parent_account` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`admin` tinyint(1) NOT NULL,
`remember_token` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`),
UNIQUE KEY `users_email_unique` (`email`)
) ENGINE=MyISAM AUTO_INCREMENT=50 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
+----+---------------------------+-------------------------------------+---------------------------------+
| id | name | email | organisation |
+----+---------------------------+-------------------------------------+---------------------------------+
| 6 | Bruce Mayert MD | Hodkiewicz.Nicolette#gmail.com | Corkery Group |
| 30 | Mr. Willard Bogisich III | Rowena13#gmail.com | Corkery Group |
| 31 | Jacinthe Murphy | Schuyler57#Pfeffer.org | Corkery Group |
| 32 | Zelda Koss PhD | iTillman#Spinka.biz | Corkery Group |
| 33 | Mr. Kevon McCullough MD | kMarks#Green.org | Corkery Group |
| 34 | Prof. Cleveland Prohaska | Ibrahim.Schneider#hotmail.com | Corkery Group |
As you can see, multiple users belong to an organisation. Now if I ran the following query:
SELECt ticket_id from tickets where organisation = 'Corkery Group';
I receive the following:
80 rows in set (0.00 sec)
From the query, I am wanting to get the organisation name and count all the tickets that belong to an organisation.
When I run the original query, it returns a result set of 480 rows when it should only return 80 rows.
There was a very good answer here Why do the results of this MySQL query get multiplied by each other? that put me in the right direction.
In my scenario, for every user with an organisation the COUNT() would be executed. The issue is, if I have 6 users which are part of one organisation, it's going to run the COUNT() statement six times.
So going back to my query, I didn't need the users table. It was irrelevant because I was only trying to access organisations with tickets & the organisation name - all of which are stored in the tickets & tickets_statuses tables. Which left me with a working query:
return DB::table('tickets')
->leftjoin('tickets_statuses', 'tickets.ticket_id', '=', 'tickets_statuses.ticket_id')
->select('tickets.organisation', DB::raw('COUNT(tickets.ticket_id) as tickets'))
->where('tickets_statuses.status_id', 0)
->groupby('tickets.organisation')
->havingRaw('COUNT(tickets.ticket_id) > 1')
->get();
Thanks to everyone who replied!

Should the following queries take 1 minute or more to complete?

The following queries use 80% or more CPU and can take more than 1 minute to complete.
My question: Is there anything wrong with my queries that would cause CPU usage like that? Can I decrease CPU usage and query time by optimizing the MySQL server conf?
Query 1 (loan_history contains 2.6 million records)
SELECT officer, SUM(balance) as balance
FROM loan_history
WHERE bank_id = '1'
AND date ='2013-07-04'
AND officer IS NOT NULL
AND officer <> ''
GROUP BY officer
ORDER BY officer;
Query 2 (loan_history contains 2.6 million records)
SELECT SUM(weighted_interest_rate) as total
FROM (SELECT balance, tmp1.balance_sum,
(balance / tmp1.balance_sum * interest_rate) as weighted_interest_rate
FROM loan_history,
(SELECT SUM(balance) balance_sum FROM loan_history
WHERE date = '2013-07-04'
AND bank_id = '1') as tmp1
WHERE date = '2013-07-04'
AND bank_id = '1') tmp2
Table information:
CREATE TABLE `loan_history` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`bank_id` int(11) DEFAULT NULL,
`loan_purpose_id` int(11) DEFAULT NULL,
`date` date NOT NULL,
`credit_grade` varchar(5) COLLATE utf8_unicode_ci DEFAULT NULL,
`interest_rate` decimal(5,2) NOT NULL,
`officer` varchar(5) COLLATE utf8_unicode_ci DEFAULT NULL,
`balance` decimal(10,2) NOT NULL,
`start_date` date DEFAULT NULL,
`days_delinquent` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `IDX_9F5FE3F11C8FB41` (`bank_id`),
KEY `IDX_9F5FE3F6F593857` (`loan_purpose_id`),
KEY `date` (`date`),
KEY `credit_grade` (`credit_grade`),
KEY `officer` (`officer`),
KEY `start_date` (`start_date`),
KEY `days_delinquent` (`days_delinquent`),
KEY `interest_rate` (`interest_rate`),
KEY `balance` (`balance`),
CONSTRAINT `FK_9F5FE3F11C8FB41` FOREIGN KEY (`bank_id`) REFERENCES `bank` (`id`),
CONSTRAINT `FK_9F5FE3F6F593857` FOREIGN KEY (`loan_purpose_id`) REFERENCES `loan_purpose` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2630634 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Query 1 EXPLAIN:
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
| 1 | SIMPLE | loan_history | index_merge | IDX_9F5FE3F11C8FB41,date,officer | date,IDX_9F5FE3F11C8FB41 | 3,5 | NULL | 4829 | Using intersect(date,IDX_9F5FE3F11C8FB41); Using where; Using temporary; Using filesort |
Query 2 EXPLAIN:
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 8236 |
| 2 | DERIVED | <derived3> | system | NULL | NULL | NULL | NULL | 1 |
| 2 | DERIVED | loan_history | index_merge | IDX_9F5FE3F11C8FB41,date | date,IDX_9F5FE3F11C8FB41 | 3,5 | NULL | 4829 | Using intersect(date,IDX_9F5FE3F11C8FB41); Using where; Using index |
| 3 | DERIVED | loan_history | index_merge | IDX_9F5FE3F11C8FB41,date | date,IDX_9F5FE3F11C8FB41 | 3,5 | NULL | 4829 | Using intersect(date,IDX_9F5FE3F11C8FB41); Using where; Using index |
My.cnf file:
default-storage-engine=MyISAM
interactive_timeout=300
key_buffer_size=256M
key_cache_block_size=4096
max_heap_table_size=128M
max_join_size=1000000000
max_allowed_packet=32M
open_files_limit=4096
query_cache_size=256M
query_cache_limit=10240M
query_cache_type=1
table_cache=256
thread_cache_size=100
tmp_table_size=128M
wait_timeout=7800
max_user_connections=50
join_buffer_size=256K
sort_buffer_size=4M
read_rnd_buffer_size=1M
innodb_open_files=300
innodb_log_file_size=256M
innodb_log_buffer_size=8M
innodb_file_per_table=1
innodb_additional_mem_pool_size=20M
innodb_flush_log_at_trx_commit=0
innodb_flush_method=O_DIRECT
innodb_support_xa=0
innodb_thread_concurrency=0
innodb_buffer_pool_size=3000M
The sum() and GROUP BY from the first query could be taking some time but I don't think there is much you can do there.
In the second query your FROM (SELECT.... is probably hitting the system pretty hard, I would recommend turning
(SELECT balance, tmp1.balance_sum,
(balance / tmp1.balance_sum * interest_rate) as weighted_interest_rate
FROM loan_history,
(SELECT SUM(balance) balance_sum FROM loan_history
WHERE date = '2013-07-04'
AND bank_id = '1') as tmp1
WHERE date = '2013-07-04'
AND bank_id = '1')
into a view or figuring out how to do it with JOINs
Please tell us how much does exactly each one take. More than 1 minute each isn't that much indicative.
Any how, From MySQL manual
When tuning a MySQL server, the two most important variables to configure are key_buffer_size and table_cache. You should first feel confident that you have these set appropriately before trying to change any other variables.
Also, take a look here.
As for optimization, first try a composite index.

Thoughts on improving these queries (MySQL) ? My production machine is seeing 10 sec times on them (using a sample size of over 1000)

Table definition and queries explained:
item |
CREATE TABLE `item` (
`item_id` int(11) NOT NULL AUTO_INCREMENT,
`item_type_id` int(11) NOT NULL,
`brand_id` int(11) NOT NULL,
`site_id` int(11) NOT NULL,
`seller_id` int(11) NOT NULL,
`title` varchar(175) NOT NULL,
`desc` text NOT NULL,
`url` varchar(767) NOT NULL,
`price` int(11) NOT NULL,
`photo` varchar(255) NOT NULL,
`photo_file` varchar(255) NOT NULL,
`photo_type` varchar(32) NOT NULL,
`has_photo` enum('yes','no','pending') NOT NULL DEFAULT 'pending',
`added_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`updated_at` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`created_at` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`normalized_time` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`location` varchar(128) NOT NULL,
PRIMARY KEY (`item_id`),
KEY `item_type_id` (`item_type_id`),
KEY `brand_id` (`brand_id`),
KEY `site_id` (`site_id`),
KEY `seller_id` (`seller_id`),
KEY `created_at` (`created_at`),
KEY `added_at` (`added_at`),
KEY `normalized_time` (`normalized_time`),
KEY `typephototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `brandidphoto` (`brand_id`,`item_type_id`,`has_photo`),
KEY `brandidphoto2` (`brand_id`,`item_type_id`,`has_photo`),
KEY `idphoto` (`item_type_id`,`has_photo`),
KEY `idphototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `idphoto2` (`item_type_id`,`has_photo`),
KEY `typepricebrandid` (`item_type_id`,`price`,`brand_id`,`item_id`),
KEY `sellertypephototime` (`seller_id`,`item_type_id`,`has_photo`,`normalized_time`),
KEY `typephoto` (`item_type_id`,`has_photo`)
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1 |
mysql> explain SELECT item.* FROM item WHERE item.item_type_id = "1" AND item.has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
| 1 | SIMPLE | item | ref | item_type_id,typephototime,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | typephototime | 5 | const,const | 69528 | Using where |
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
1 row in set (0.02 sec)
mysql> explain SELECT * FROM item WHERE item_type_id = "1" AND (price BETWEEN "25" AND "275") AND brand_id = "10" ORDER BY item_id DESC LIMIT 1;
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | item | index | item_type_id,brand_id,typephototime,brandidphoto,brandidphoto2,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | PRIMARY | 4 | NULL | 203 | Using where |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
1 row in set (0.01 sec)
mysql> explain SELECT item.* FROM item WHERE item.brand_id = "10" AND item.item_type_id = "1" AND item.has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
| 1 | SIMPLE | item | index | item_type_id,brand_id,typephototime,brandidphoto,brandidphoto2,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | normalized_time | 8 | NULL | 502397 | Using where |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
1 row in set (2.15 sec)
mysql> explain SELECT COUNT(*) FROM item WHERE item.item_type_id = "1" AND item.has_photo = "yes" ;
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
| 1 | SIMPLE | item | ref | item_type_id,typephototime,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | typephoto | 5 | const,const | 71135 | Using where; Using index |
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
1 row in set (0.01 sec)
The following indexes are redundant because they match the left columns of another index. You can almost certainly drop these indexes and save some space and overhead.
KEY `item_type_id` (`item_type_id`), /* redundant */
KEY `brand_id` (`brand_id`), /* redundant */
KEY `seller_id` (`seller_id`), /* redundant */
KEY `idphototime` (`item_type_id`,`has_photo`,`normalized_time`), /* redundant */
KEY `brandidphoto2` (`brand_id`,`item_type_id`,`has_photo`), /* redundant */
KEY `idphoto` (`item_type_id`,`has_photo`), /* redundant */
KEY `idphoto2` (`item_type_id`,`has_photo`), /* redundant */
KEY `typephoto` (`item_type_id`,`has_photo`) /* redundant */
That leaves the following indexes:
KEY `site_id` (`site_id`),
KEY `created_at` (`created_at`),
KEY `added_at` (`added_at`),
KEY `normalized_time` (`normalized_time`),
KEY `brandidphoto` (`brand_id`,`item_type_id`,`has_photo`),
KEY `typephototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `typepricebrandid` (`item_type_id`,`price`,`brand_id`,`item_id`),
KEY `sellertypephototime` (`seller_id`,`item_type_id`,`has_photo`,`normalized_time`),
You can also use a tool like pt-duplicate-key-checker to find redundant indexes.
Next consider the storage engine:
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1;
Almost always, InnoDB is a better choice than MyISAM. Not only for performance, but for data integrity and crash safety. InnoDB has been the default storage engine since 2010, and it's the only storage engine that is actively getting improved. I'd recommend making a copy of this table, changing the storage engine to InnoDB, and compare its performance with respect to your queries.
Next let's consider indexes for the queries:
SELECT item.* FROM `item` WHERE item.item_type_id = "1" AND item.has_photo = "yes"
ORDER BY normalized_time DESC LIMIT 1;
I would choose an index on (item_type_id, has_photo, normalized_time) and that's the index it's currently using, which is typephototime.
One way to optimize this further would be to fetch only the columns in the index. That's when you see "Using index" in the EXPLAIN plan, it can be a huge improvement for performance.
Another important factor is to make sure that your index is cached in memory: increase key_buffer_size if you use MyISAM or innodb_buffer_pool_size if you use InnoDB to be as large as all the indexes you want to remain in memory. Because you don't want to run a query that needs to scan an index larger than your buffers; it causes a lot of swapping.
SELECT * FROM `item` WHERE item_type_id = "1" AND (price BETWEEN "25" AND "275") AND brand_id = "10"
ORDER BY item_id DESC LIMIT 1;
I would choose an index on (item_type_id, brand_id, price), but this query is currently using the PRIMARY index. You should create a new index.
SELECT item.* FROM `item` WHERE item.brand_id = "10" AND item.item_type_id = "1" AND item.has_photo = "yes"
ORDER BY normalized_time DESC LIMIT 1;
I would choose an index on (item_type_id, brand_id, has_photo, normalized_time). You should create a new index.
SELECT COUNT(*) FROM `item` WHERE item.item_type_id = "1" AND item.has_photo = "yes" ;
I would choose an index on (item_type_id, has_photo) and that's the index it's currently using, which is typephoto. It's also getting the "Using index" optimization, so the only other improvement could be to make sure there's enough buffer to hold the index in memory.
It's hard to optimize SELECT COUNT(*) queries because they naturally have to scan a lot of rows.
The other strategy to optimize COUNT(*) is to calculate the counts offline, and store them either in a summary table or in an in-memory cache like memcached so you don't have to recalculate them every time someone loads a page. But that means you have to update the counts every time someone adds or deletes a row in the item table, which could be more costly depending on how frequently that happens.
A few things I would suggest changing:
You don't need all those indexes. You really only need indexes on fields that are accessed a lot, like foreign key fields. Remove all the indexes except ones on ID fields.
You should be storing dates as nulls unless there is actual data.
Stay away from the enum data type, use smallint with flags representing each value. Example, 0 pending, 1 yes, 2 no.
Alongside reducing the size of the database, it makes things much cleaner. Your new table structure would look like so:
CREATE TABLE `item` (
`item_id` int(11) NOT NULL AUTO_INCREMENT,
`item_type_id` int(11) NOT NULL,
`brand_id` int(11) NOT NULL,
`site_id` int(11) NOT NULL,
`seller_id` int(11) NOT NULL,
`title` varchar(175) NOT NULL,
`desc` text NOT NULL,
`url` varchar(767) NOT NULL,
`price` int(11) NOT NULL,
`photo` varchar(255) NOT NULL,
`photo_file` varchar(255) NOT NULL,
`photo_type` varchar(32) NOT NULL,
`has_photo` smallint NOT NULL DEFAULT 0,
`added_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`updated_at` datetime NULL DEFAULT NULL,
`created_at` datetime NULL DEFAULT NULL,
`normalized_time` datetime NULL DEFAULT NULL,
`location` varchar(128) NULL,
PRIMARY KEY (`item_id`),
KEY `item_type_id` (`item_type_id`),
KEY `brand_id` (`brand_id`),
KEY `site_id` (`site_id`),
KEY `seller_id` (`seller_id`)
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1;
I would also suggest using utf8_unicode_ci as the collate, utf8 as the charset, and InnoDB as the engine.
But first off, remove all those keys and try again. Also remove the aliasing on the 3rd query.
SELECT * FROM item WHERE brand_id = "10" AND item_type_id = "1" AND has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;

Mysql 5.6 doesn't want to set a column to NULL

I am about 98% sure this query has been working properly on mysql 5.1. We have upgraded one of our machines to mysql 5.6 and I am running an update and its getting a database error with this query:
UPDATE diagnostic
LEFT JOIN contact ON diagnosticdata_suppliercontact = contact_id
SET diagnosticdata_suppliercontact = NULL
WHERE (!contactdata_issupplier) OR (contact.contact_id IS NULL);
However the table schema allows it to be NULL. Does anyone know of any problems or changes that happened in mysql 5.6 that could cause us grief?
Here is a description of the table:
mysql> describe diagnostic;
+--------------------------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------------------+-------------+------+-----+---------+----------------+
| diagnostic_id | int(11) | NO | PRI | NULL | auto_increment |
| diagnostic_time | int(15) | NO | | NULL | |
| diagnostic_user | int(15) | NO | | NULL | |
| diagnosticdata_time | int(15) | NO | | NULL | |
| diagnosticdata_user | int(15) | NO | | NULL | |
| diagnosticdata_name | varchar(50) | NO | | NULL | |
| diagnosticdata_suppliercontact | int(11) | YES | MUL | NULL | |
+--------------------------------+-------------+------+-----+---------+----------------+
7 rows in set (0.00 sec)
ERROR 1048 (23000): Column 'diagnosticdata_suppliercontact' cannot be null
CREATE STATEMENT IS HERE TO there are triggers:
delimiter $$
CREATE TABLE `diagnostic` (
`diagnostic_id` int(11) NOT NULL AUTO_INCREMENT,
`diagnostic_time` int(15) NOT NULL,
`diagnostic_user` int(15) NOT NULL,
`diagnosticdata_time` int(15) NOT NULL,
`diagnosticdata_user` int(15) NOT NULL,
`diagnosticdata_name` varchar(50) NOT NULL,
`diagnosticdata_suppliercontact` int(11) DEFAULT NULL,
PRIMARY KEY (`diagnostic_id`),
KEY `diagnostic_suppliercontact` (`diagnosticdata_suppliercontact`) USING BTREE,
CONSTRAINT `fk_diagnosticdata_suppliercontact_contact` FOREIGN KEY (`diagnosticdata_suppliercontact`) REFERENCES `contact` (`contact_id`)
) ENGINE=InnoDB AUTO_INCREMENT=188 DEFAULT CHARSET=utf8$$
CREATE
DEFINER=`dotuser`#`localhost`
TRIGGER `ezymerged`.`diagnostic_insert`
AFTER INSERT ON `ezymerged`.`diagnostic`
FOR EACH ROW
BEGIN
REPLACE INTO ezymerged_history.diagnostic
SELECT diagnostic.* FROM ezymerged.diagnostic
WHERE diagnostic_id=NEW.diagnostic_id;
END
$$
CREATE
DEFINER=`dotuser`#`localhost`
TRIGGER `ezymerged`.`diagnostic_update`
AFTER UPDATE ON `ezymerged`.`diagnostic`
FOR EACH ROW
BEGIN
REPLACE INTO ezymerged_history.diagnostic
SELECT diagnostic.* FROM ezymerged.diagnostic
WHERE diagnostic_id=NEW.diagnostic_id;
END
$$
While the diagnosticdata_suppliercontact allows for NULLs the constraint:
CONSTRAINT `fk_diagnosticdata_suppliercontact_contact`
FOREIGN KEY (`diagnosticdata_suppliercontact`)
REFERENCES `contact` (`contact_id`)
will take precedence, and requires the value in diagnosticdata_suppliercontact to have a matching value in the contact_id field in the contact table.

Optimize this query further? - indexed but still slow

I have a query below that joins 4 tables. I have added indexes and the explain output looks good with indexes used (see below).
Can I optimize the query further?
The modelXml is rather big on some records.For a big project where I get back 22 records, each with approximately 2.5 - 3MB of modelXml data, the query is taking long (total of 69MB of data returned).
I suspect this is the issue but not sure how to deal with it.
I was reading on adjusting internal mysql variables e.g key_buffer_size and table_cache. Would any of this help?
key_buffer_size is currently set at 8384512 (~8MB) and table_cache at 64
What should I increase it to?
What other variables should I be looking at to manage to speed up return of such big data?
Any other suggestions are welcome. I am a novice to mysql but really trying to get better.
SELECT `m`.`modelId`, `m`.`modelTypeId`, `m`.`modelXml`, `m`.`xmlSize`, `m`.`createdById`, `m`.`creationDate`, `m`.`modifiedDate`, `u`.`firstName`, `u`.`lastName` FROM `models_1` AS `m`
INNER JOIN `modelFolderAssociations_1` AS `mfa` ON m.modelId = mfa.modelIOId
INNER JOIN `modelFolders_1` AS `mf` ON mfa.folderId = mf.folderId
INNER JOIN `users_1` AS `u` ON m.createdById = u.userId
WHERE (m.projectId = 2) AND (mfa.folderId = 5) AND (mfa.modelIOType = 2) AND (m.modelTypeId = 2)
CREATE TABLE `models` (
`modelId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`groupId` int(11) NOT NULL,
`projectId` int(11) NOT NULL,
`createdById` int(11) NOT NULL,
`modelTypeId` int(11) NOT NULL,
`modelXml` longtext,
`modelSpecXml` longtext NOT NULL,
`xmlSize` bigint(20) NOT NULL default '0',
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`modelId`,`customerId`)
) ENGINE=InnoDB AUTO_INCREMENT=75 DEFAULT CHARSET=utf8
CREATE TABLE `modelFolders` (
`folderId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`groupId` int(11) NOT NULL,
`projectId` int(11) NOT NULL,
`parentId` int(11) NOT NULL,
`folderName` varchar(64) NOT NULL,
`folderType` int(11) NOT NULL,
`editable` tinyint(1) NOT NULL default '1',
`nextDefaultNameNumber` int(11) NOT NULL default '1',
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`folderId`,`customerId`),
KEY `parentId` (`parentId`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=utf8
CREATE TABLE `modelFolderAssociations` (
`associationId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`folderId` int(11) NOT NULL,
`projectId` int(11) NOT NULL,
`modelIOId` int(11) NOT NULL,
`modelIOType` tinyint(1) NOT NULL default '1',
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`associationId`,`customerId`),
KEY `folderId` (`folderId`,`modelIOType`)
) ENGINE=InnoDB AUTO_INCREMENT=75 DEFAULT CHARSET=utf8
CREATE TABLE `users` (
`userId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`userName` varchar(50) NOT NULL,
`password` varchar(256) NOT NULL,
`firstName` varchar(50) default NULL,
`lastName` varchar(50) default NULL,
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`userId`,`customerId`),
UNIQUE KEY `userName` (`userName`)
) ENGINE=InnoDB AUTO_INCREMENT=21 DEFAULT CHARSET=utf8
Explain output
+----+-------------+-------------------------+--------+---------------+----------+---------+---------------------------------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------------+--------+---------------+----------+---------+---------------------------------------------------+------+-------------+
| 1 | SIMPLE | modelFolders | const | PRIMARY | PRIMARY | 8 | const,const | 1 | Using index |
| 1 | SIMPLE | modelFolderAssociations | ref | folderId | folderId | 5 | const,const | 22 | Using where |
| 1 | SIMPLE | models | eq_ref | PRIMARY | PRIMARY | 8 | xa_system.modelFolderAssociations.modelIOId,const | 1 | Using where |
| 1 | SIMPLE | users | eq_ref | PRIMARY | PRIMARY | 8 | xa_system.models.createdById,const | 1 | |
+----+-------------+-------------------------+--------+---------------+----------+---------+---------------------------------------------------+------+-------------+
Having large text columns, like those you're using to store XML I presume, can hurt performance regardless of how well your indexes are structured.
It can be better in these cases to move the text columns to a separate table, indexed by string length which you're already storing and CRC32.
CREATE TABLE MODEL_XML (
xmlId INT(11) unsigned NOT NULL auto_increment,
xmlSize BIGINT(20) NOT NULL default '0',
crc32 INT(11) unsigned NOT NULL,
xmlData LONGTEXT,
PRIMARY KEY (xmlId),
UNIQUE KEY (xmlSize, crc32)
)
Then the width of your columns on the table with your important indices becomes constant.
ex.
modelXmlId INT(11) unsigned NOT NULL
specXmlId INT(11) unsigned NOT NULL
It also has the benefit of being more space-efficient for redundant text (empty strings, etc) since they would all share one xmlId and thus one row in the DB.
You should index your foreign keys, this will really help with your joins:
CREATE INDEX IDX_MODELS_CUSTID
on models (customerId)
CREATE INDEX IDX_FLDR_ASSOC_MODELIO
modelFolderAssociations(modelIOId)
CREATE INDEX IDX_FLDR_ASSOC_FLDRID
modelFolderAssociations(folderId)
and so on.
Test your query after taking off the modelXml column from the SELECT clause.
If the speed is then significantly better, then the slowness comes from the volume off data to transfer, not the query itself.
Your indexes don't always match the queries and joins. If all of the columns in your WHERE and JOIN clauses are not in the index, you're forcing mysql to look at the underlying row, which will kill your performance, especially since your model rows are so wide.
For "modelFolderAssociations", you made the correct composite key for the where clause, but you should include modelIOId for the join to model.
For "models", you'd want a composite index on (modelId, projectId, modelTypeId, createdById ) to cover the incoming link from mfa, the two items in the where clause, and the outbound link to users.
For "modelFolders" and "users", you have the incoming join covered in your primary keys.
The engine is only going to use one index, so adding extra index single indexes (as Mike suggested) won't be as good.

Categories