Related
I have some trouble with the execution time of a script. The query takes a lot of time. This is the query:
select avg(price) from voiture where duration<30 AND make="Audi" AND model="A4"
+---+---+---+---+---+---+---+---+---+---+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+---+---+---+---+---+---+---+---+---+---+
| 1 | SIMPLE | voiture | ALL | NULL | NULL | NULL | NULL | 1376949 | Using where |
+---+---+---+---+---+---+---+---+---+---+
select price from voiture where duration<30 AND make="Audi" AND model="A4"
+---+---+---+---+---+---+---+---+---+---+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+---+---+---+---+---+---+---+---+---+---+
| 1 | SIMPLE | voiture | ALL | NULL | NULL | NULL | NULL | 1376949 | Using where |
+---+---+---+---+---+---+---+---+---+---+
This query take around 2 seconds to be executed on the phpMyAdmin interface. I tried to see what the issue was and removing the avg function makes the query lasts around 0.0080 seconds.
I asked myself how long it would take to make the calculate the avg in the php script, but the query with or withouth avg takes around 2 seconds both.
So I decided to take all the values of my table and make the process in the script, so I use this query:
select * from voiture where duration<30 AND make="Audi" AND model="A4"
+---+---+---+---+---+---+---+---+---+---+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+---+---+---+---+---+---+---+---+---+---+
| 1 | SIMPLE | voiture | ALL | NULL | NULL | NULL | NULL | 1376949 | Using where |
+---+---+---+---+---+---+---+---+---+---+
On the phpMyAdmin interface, it takes 0.0112 seconds. But in my php script, it takes 25 seconds !
$timestart2=microtime(true);
$querysec='select * from voiture where duration<30 AND make="Audi" AND model="A4"';
$requestsec=mysql_query($querysec) or die(mysql_error());
$timeend2=microtime(true);
$time2=$timeend2-$timestart2;
$page_load_time2 = number_format($time2, 9);
Here is the table structure:
CREATE TABLE `voiture` (
`carid` bigint(9) NOT NULL,
`serviceid` bigint(9) NOT NULL,
`service` varchar(256) NOT NULL,
`model` varchar(256) DEFAULT NULL,
`gearingType` varchar(256) DEFAULT NULL,
`displacement` int(5) DEFAULT NULL,
`cylinders` int(2) DEFAULT NULL,
`fuel` varchar(32) DEFAULT NULL,
`mileage` int(7) DEFAULT NULL,
`existFlag` tinyint(1) DEFAULT NULL,
`lastUpdate` date DEFAULT NULL,
`version` varchar(256) DEFAULT NULL,
`bodyType` varchar(256) DEFAULT NULL,
`firstRegistration` date DEFAULT NULL,
`powerHp` int(4) DEFAULT NULL,
`powerKw` int(4) DEFAULT NULL,
`vat` varchar(256) DEFAULT NULL,
`price` decimal(12,2) DEFAULT NULL,
`duration` int(3) DEFAULT NULL,
`pageUrl` varchar(256) DEFAULT NULL,
`carImg` varchar(256) DEFAULT NULL,
`color` varchar(256) DEFAULT NULL,
`doors` int(1) DEFAULT NULL,
`seats` int(1) DEFAULT NULL,
`prevOwner` int(1) DEFAULT NULL,
`co2` varchar(256) DEFAULT NULL,
`consumption` varchar(256) DEFAULT NULL,
`gears` int(1) DEFAULT NULL,
`equipment` varchar(1024) NOT NULL,
`make` varchar(256) NOT NULL,
`country` varchar(3) NOT NULL
)
There's an index on carid and serviceid
Why does my query takes so long to be executed ? Is there a way it can be improved ?
Why is the execution time different from phpMyAdmin and my php script ?
On the phpMyAdmin interface, it takes 0.0112 seconds. But in my php
script, it takes 25 seconds!
phpMyAdmin interface adds LIMIT to each query. By default it's LIMIT 30.
To decrease time of your aggregate query you need to create indexes for each condition you use(or one composite index, may be).
So, try to create indexes for your model, make and duration fields.
Also, your table is too denormalized. You can to create pair of table to normalize it for a bit.
Ex: Vendors(id, name), Models(id, name) and modify your voiture to have vendor_id/model_id fields instead of text make/model.
Then your initial query will look like:
select avg(t.price) from voiture t
INNER JOIN Models m ON m.id = t.model_id
INNER JOIN Vendors v ON v.id = t.vendor_id
where t.duration<30 AND v.name="Audi" AND m.name="A4"
It will scan light lookup tables for text matches and operate with your heavy table with indexed ids.
There are many possible solution, first thing you can do is create index at DB level this can also improve your execution time.
second this check server, there may be some processes which would occupying your server resources, and making your server slow.
You can make it faster using GROUP BY like
select AVG(price) from voiture where duration<30 AND make="Audi" AND model="A4" GROUP BY make
Also you can add an index for make column
ALTER TABLE `voiture` ADD INDEX `make` (`make`)
My MySQL app allows users to create bookmarks and organize them with tags.
A tag is added to a bookmark with a middle MySQL DB table bookmark_tag_relationship
I would like to add a Foreign Key to auto-delete all records in bookmark_tag_relationship when a Bookmarks record is deleted.
All the bookmark_tag_relationship records deleted would have a bookmark_id column that matches the id column on the bookmark record that is deleted
How can I set this up correctly based on the 3 table structures shown below?
Failed attempt #1
I tried something like this:
ALTER TABLE bookmarks ADD FOREIGN KEY (`id`)
REFERENCES bookmark_tag_relationship(`bookmark_id`);
It returned an error with no information just that it could not create it.
table bookmarks
--
-- Table structure for table `bookmarks`
--
CREATE TABLE IF NOT EXISTS `bookmarks` (
`id` int(20) NOT NULL,
`url_id` int(20) DEFAULT NULL,
`user_id` int(30) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`url` varchar(255) DEFAULT NULL,
`favicon_url` varchar(255) DEFAULT NULL,
`project_url` varchar(255) DEFAULT NULL,
`github_url` varchar(255) DEFAULT NULL,
`demo_url` int(255) DEFAULT NULL,
`local_demo_url` varchar(255) DEFAULT NULL,
`image1` varchar(255) DEFAULT NULL,
`image2` varchar(255) DEFAULT NULL,
`image3` varchar(255) DEFAULT NULL,
`image4` varchar(255) DEFAULT NULL,
`description` text,
`notes` text,
`tags_string` varchar(255) DEFAULT NULL,
`click_count` int(10) NOT NULL DEFAULT '0',
`tag_count` int(10) NOT NULL DEFAULT '0',
`created_on` datetime DEFAULT NULL,
`updated_on` datetime DEFAULT NULL,
`last_viewed_on` datetime DEFAULT NULL,
`active` int(1) NOT NULL DEFAULT '1'
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
table tags
--
-- Table structure for table `tags`
--
CREATE TABLE IF NOT EXISTS `tags` (
`id` int(11) NOT NULL,
`user_id` int(11) DEFAULT NULL,
`name` varchar(100) DEFAULT NULL,
`description` text,
`color` varchar(10) DEFAULT NULL,
`bookmark_count` int(11) NOT NULL DEFAULT '0',
`active` int(11) NOT NULL DEFAULT '1'
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latin1;
table bookmark_tag_relationship
--
-- Table structure for table `bookmark_tag_relationship`
--
CREATE TABLE IF NOT EXISTS `bookmark_tag_relationship` (
`id` int(11) NOT NULL,
`bookmark_id` int(30) NOT NULL,
`tag_id` int(30) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Three issues:
You have not defined a primary key in the table bookmarks (or any other);
In comments you clarified you actually have them, so then this is a non-issue;
You have the foreign key constraint defined in the opposite direction. You should define it on the child table;
You need to add the ON DELETE CASCADE clause.
So, execute this, and it will work (you can skip the first one, since in comments you clarified you already have the primary key):
ALTER TABLE bookmarks ADD
CONSTRAINT PRIMARY KEY(id);
ALTER TABLE bookmark_tag_relationship
ADD FOREIGN KEY (bookmark_id)
REFERENCES bookmarks(id)
ON DELETE CASCADE;
Now you can delete a bookmark, for example:
DELETE FROM bookmarks WHERE id = 1;
...and the related records in bookmark_tag_relationship will be deleted in the same transaction.
Some additional remarks:
The difference between int(20) and int(30) that you have for the primary key and foreign key types cannot be very useful. It does not influence the constraint, but I would still suggest to harmonise that as follows:
ALTER TABLE bookmark_tag_relationship
MODIFY bookmark_id int(20) NOT NULL;
It is not needed to surround your table and column names with backticks. Only when you have chosen reserved words for those names, they are needed, but that is something you'd want to avoid anyway.
EDIT: You're foreign key is also failing because you do not have a primary key. A foreign key is a reference to a primary key.
I would setup a trigger for bookmarks on delete.
DELIMITER $$
CREATE DEFINER=`username`#`%` TRIGGER `bookmarks_ADEL` AFTER DELETE ON `bookmarks` FOR EACH ROW
BEGIN
DELETE FROM bookmark_tag_relationship WHERE bookmark_id = old.id;
END
When a row from Bookmarks is deleted, it will take that ID and remove the row out of bookmark_tag_relationship.
You need to set Primary keys in both those tables, most likely the ID
One note about using a cascade delete on foreign key. It will not activate any triggers that are set on those deleted rows. So as you progress, if you decide to trigger off them to remove or modify more information, the foreign key cascade will not activate them and you'll have to trigger on delete anyway.
Try to add a ON DELETE CASCADE on your foreign key.
ALTER TABLE bookmark_tag_relationship ADD FOREIGN KEY (`bookmark_id`)
REFERENCES bookmarks(`id`)
ON DELETE CASCADE;
There's a few reasons why your foreign key is not working.
First of all, you need to make sure the column id of each table is a primary key.
ALTER TABLE table_name
ADD PRIMARY KEY (table_column)
After you've added your primary key, you need to make sure that their type is the same.
In your bookmark_tag_relationship table, the bookmark_id is INT(30), but in your bookmarks* table, your column 'id' is **INT(20). You need to make sure both of them are the same. So either change the id from bookmarks to INT(30) or change the other one to INT(20).
And at last, your foreign key query is reversed. You want it the other way around.
ALTER TABLE bookmark_tag_relationship ADD FOREIGN KEY (`bookmark_id`)
REFERENCES bookmarks(`id`)
ON DELETE CASCADE;
You want the bookmark_tag_relationship to be deleted when a bookmark is deleted, and not the other way around.
Use a primary auto incremented key on the id columns. And then like others said use cascade on delete with foreign keys.
mysql> show create table bookmarks\G
*************************** 1. row ***************************
Table: bookmarks
Create Table: CREATE TABLE `bookmarks` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
mysql> show create table tags\G
*************************** 1. row ***************************
Table: tags
Create Table: CREATE TABLE `tags` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
mysql> show create table bookmarks_tags\G
*************************** 1. row ***************************
Table: bookmarks_tags
Create Table: CREATE TABLE `bookmarks_tags` (
`bookmark_id` int(11) NOT NULL DEFAULT '0',
`tag_id` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`bookmark_id`,`tag_id`),
CONSTRAINT `bookmarks_tags_ibfk_1` FOREIGN KEY (`bookmark_id`) REFERENCES `bookmarks` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.01 sec)
mysql> select * from bookmarks;
+----+-----------+
| id | title |
+----+-----------+
| 1 | bookmark1 |
| 2 | bookmark2 |
| 3 | bookmark3 |
+----+-----------+
3 rows in set (0.00 sec)
mysql> select * from tags;
+----+------+
| id | name |
+----+------+
| 1 | tag1 |
| 2 | tag2 |
| 3 | tag3 |
| 4 | tag4 |
| 5 | tag5 |
+----+------+
5 rows in set (0.00 sec)
mysql> select * from bookmarks_tags;
+-------------+--------+
| bookmark_id | tag_id |
+-------------+--------+
| 1 | 1 |
| 1 | 2 |
| 1 | 3 |
| 2 | 1 |
| 2 | 4 |
| 3 | 5 |
+-------------+--------+
6 rows in set (0.00 sec)
mysql> delete from bookmarks where title = 'bookmark1';
Query OK, 1 row affected (0.01 sec)
mysql> select * from bookmarks; select * from tags; select * from bookmarks_tags;
+----+-----------+
| id | title |
+----+-----------+
| 2 | bookmark2 |
| 3 | bookmark3 |
+----+-----------+
2 rows in set (0.00 sec)
+----+------+
| id | name |
+----+------+
| 1 | tag1 |
| 2 | tag2 |
| 3 | tag3 |
| 4 | tag4 |
| 5 | tag5 |
+----+------+
5 rows in set (0.01 sec)
+-------------+--------+
| bookmark_id | tag_id |
+-------------+--------+
| 2 | 1 |
| 2 | 4 |
| 3 | 5 |
+-------------+--------+
3 rows in set (0.00 sec)
Im creating a Rank and Requirement table for Martial Arts School.
Each student holds a rank in the martial arts. The rank name, belt color, and rank requirements are stored. Each rank will have numerous rank requirements. Each requirement is considered a requirement just for the rank at which the requirement is introduced. Every requirement is associated with a particular rank. All ranks except white belt have at least one requirement.
My ER Diagram:
Rank and Requirement ER Diagram
Rank Table:
CREATE TABLE IF NOT EXISTS `rank` (
`rank_id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`rank_nme` VARCHAR(45) NOT NULL,
PRIMARY KEY (`rank_id`))
ENGINE = InnoDB;
Output:
+------------+--------------+----------------+
| rank_id | INT(10) | AUTO_INCREMENT |
+------------+--------------+----------------+
| rank_nme | VARCHAR(45) | |
+------------+--------------+----------------+
Requirement Table:
CREATE TABLE IF NOT EXISTS `requirement` (
`req_id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`rank_id` INT UNSIGNED NOT NULL,
`req_nme` VARCHAR(45) NOT NULL,
`req_rank_nme` VARCHAR(45) NULL,
PRIMARY KEY (`req_id`),
INDEX `requirement_rank_id_idx` (`rank_id` ASC),
CONSTRAINT `requirement_rank_id_idx`
FOREIGN KEY (`rank_id`)
REFERENCES `rank` (`rank_id`)
ON DELETE RESTRICT
ON UPDATE CASCADE)
ENGINE = InnoDB;
Output:
+---------------+--------------+----------------+
| req_id | INT(10) | AUTO_INCREMENT |
+---------------+--------------+----------------+
| rank_id | INT(10) | |
+---------------+--------------+----------------+
| req_nme | VARCHAR(45) | |
+---------------+--------------+----------------+
| req_rank_nme | VARCHAR(45) | |
+---------------+--------------+----------------+
Need help if Im doing it right or wrong or you guys have modification or any suggestions! Thanks!
First in the first table you will need an id that is unique and auto increment for db good practice. Then you go with the rank_id, rank_name. If you want to associate the rank with a user, you will need another foreign key to do it. user_id from users.id .
Now you have rank_id which increments and i don't see the point of doing that. The rest is ok i think.
Table definition and queries explained:
item |
CREATE TABLE `item` (
`item_id` int(11) NOT NULL AUTO_INCREMENT,
`item_type_id` int(11) NOT NULL,
`brand_id` int(11) NOT NULL,
`site_id` int(11) NOT NULL,
`seller_id` int(11) NOT NULL,
`title` varchar(175) NOT NULL,
`desc` text NOT NULL,
`url` varchar(767) NOT NULL,
`price` int(11) NOT NULL,
`photo` varchar(255) NOT NULL,
`photo_file` varchar(255) NOT NULL,
`photo_type` varchar(32) NOT NULL,
`has_photo` enum('yes','no','pending') NOT NULL DEFAULT 'pending',
`added_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`updated_at` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`created_at` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`normalized_time` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`location` varchar(128) NOT NULL,
PRIMARY KEY (`item_id`),
KEY `item_type_id` (`item_type_id`),
KEY `brand_id` (`brand_id`),
KEY `site_id` (`site_id`),
KEY `seller_id` (`seller_id`),
KEY `created_at` (`created_at`),
KEY `added_at` (`added_at`),
KEY `normalized_time` (`normalized_time`),
KEY `typephototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `brandidphoto` (`brand_id`,`item_type_id`,`has_photo`),
KEY `brandidphoto2` (`brand_id`,`item_type_id`,`has_photo`),
KEY `idphoto` (`item_type_id`,`has_photo`),
KEY `idphototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `idphoto2` (`item_type_id`,`has_photo`),
KEY `typepricebrandid` (`item_type_id`,`price`,`brand_id`,`item_id`),
KEY `sellertypephototime` (`seller_id`,`item_type_id`,`has_photo`,`normalized_time`),
KEY `typephoto` (`item_type_id`,`has_photo`)
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1 |
mysql> explain SELECT item.* FROM item WHERE item.item_type_id = "1" AND item.has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
| 1 | SIMPLE | item | ref | item_type_id,typephototime,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | typephototime | 5 | const,const | 69528 | Using where |
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
1 row in set (0.02 sec)
mysql> explain SELECT * FROM item WHERE item_type_id = "1" AND (price BETWEEN "25" AND "275") AND brand_id = "10" ORDER BY item_id DESC LIMIT 1;
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | item | index | item_type_id,brand_id,typephototime,brandidphoto,brandidphoto2,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | PRIMARY | 4 | NULL | 203 | Using where |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
1 row in set (0.01 sec)
mysql> explain SELECT item.* FROM item WHERE item.brand_id = "10" AND item.item_type_id = "1" AND item.has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
| 1 | SIMPLE | item | index | item_type_id,brand_id,typephototime,brandidphoto,brandidphoto2,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | normalized_time | 8 | NULL | 502397 | Using where |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
1 row in set (2.15 sec)
mysql> explain SELECT COUNT(*) FROM item WHERE item.item_type_id = "1" AND item.has_photo = "yes" ;
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
| 1 | SIMPLE | item | ref | item_type_id,typephototime,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | typephoto | 5 | const,const | 71135 | Using where; Using index |
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
1 row in set (0.01 sec)
The following indexes are redundant because they match the left columns of another index. You can almost certainly drop these indexes and save some space and overhead.
KEY `item_type_id` (`item_type_id`), /* redundant */
KEY `brand_id` (`brand_id`), /* redundant */
KEY `seller_id` (`seller_id`), /* redundant */
KEY `idphototime` (`item_type_id`,`has_photo`,`normalized_time`), /* redundant */
KEY `brandidphoto2` (`brand_id`,`item_type_id`,`has_photo`), /* redundant */
KEY `idphoto` (`item_type_id`,`has_photo`), /* redundant */
KEY `idphoto2` (`item_type_id`,`has_photo`), /* redundant */
KEY `typephoto` (`item_type_id`,`has_photo`) /* redundant */
That leaves the following indexes:
KEY `site_id` (`site_id`),
KEY `created_at` (`created_at`),
KEY `added_at` (`added_at`),
KEY `normalized_time` (`normalized_time`),
KEY `brandidphoto` (`brand_id`,`item_type_id`,`has_photo`),
KEY `typephototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `typepricebrandid` (`item_type_id`,`price`,`brand_id`,`item_id`),
KEY `sellertypephototime` (`seller_id`,`item_type_id`,`has_photo`,`normalized_time`),
You can also use a tool like pt-duplicate-key-checker to find redundant indexes.
Next consider the storage engine:
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1;
Almost always, InnoDB is a better choice than MyISAM. Not only for performance, but for data integrity and crash safety. InnoDB has been the default storage engine since 2010, and it's the only storage engine that is actively getting improved. I'd recommend making a copy of this table, changing the storage engine to InnoDB, and compare its performance with respect to your queries.
Next let's consider indexes for the queries:
SELECT item.* FROM `item` WHERE item.item_type_id = "1" AND item.has_photo = "yes"
ORDER BY normalized_time DESC LIMIT 1;
I would choose an index on (item_type_id, has_photo, normalized_time) and that's the index it's currently using, which is typephototime.
One way to optimize this further would be to fetch only the columns in the index. That's when you see "Using index" in the EXPLAIN plan, it can be a huge improvement for performance.
Another important factor is to make sure that your index is cached in memory: increase key_buffer_size if you use MyISAM or innodb_buffer_pool_size if you use InnoDB to be as large as all the indexes you want to remain in memory. Because you don't want to run a query that needs to scan an index larger than your buffers; it causes a lot of swapping.
SELECT * FROM `item` WHERE item_type_id = "1" AND (price BETWEEN "25" AND "275") AND brand_id = "10"
ORDER BY item_id DESC LIMIT 1;
I would choose an index on (item_type_id, brand_id, price), but this query is currently using the PRIMARY index. You should create a new index.
SELECT item.* FROM `item` WHERE item.brand_id = "10" AND item.item_type_id = "1" AND item.has_photo = "yes"
ORDER BY normalized_time DESC LIMIT 1;
I would choose an index on (item_type_id, brand_id, has_photo, normalized_time). You should create a new index.
SELECT COUNT(*) FROM `item` WHERE item.item_type_id = "1" AND item.has_photo = "yes" ;
I would choose an index on (item_type_id, has_photo) and that's the index it's currently using, which is typephoto. It's also getting the "Using index" optimization, so the only other improvement could be to make sure there's enough buffer to hold the index in memory.
It's hard to optimize SELECT COUNT(*) queries because they naturally have to scan a lot of rows.
The other strategy to optimize COUNT(*) is to calculate the counts offline, and store them either in a summary table or in an in-memory cache like memcached so you don't have to recalculate them every time someone loads a page. But that means you have to update the counts every time someone adds or deletes a row in the item table, which could be more costly depending on how frequently that happens.
A few things I would suggest changing:
You don't need all those indexes. You really only need indexes on fields that are accessed a lot, like foreign key fields. Remove all the indexes except ones on ID fields.
You should be storing dates as nulls unless there is actual data.
Stay away from the enum data type, use smallint with flags representing each value. Example, 0 pending, 1 yes, 2 no.
Alongside reducing the size of the database, it makes things much cleaner. Your new table structure would look like so:
CREATE TABLE `item` (
`item_id` int(11) NOT NULL AUTO_INCREMENT,
`item_type_id` int(11) NOT NULL,
`brand_id` int(11) NOT NULL,
`site_id` int(11) NOT NULL,
`seller_id` int(11) NOT NULL,
`title` varchar(175) NOT NULL,
`desc` text NOT NULL,
`url` varchar(767) NOT NULL,
`price` int(11) NOT NULL,
`photo` varchar(255) NOT NULL,
`photo_file` varchar(255) NOT NULL,
`photo_type` varchar(32) NOT NULL,
`has_photo` smallint NOT NULL DEFAULT 0,
`added_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`updated_at` datetime NULL DEFAULT NULL,
`created_at` datetime NULL DEFAULT NULL,
`normalized_time` datetime NULL DEFAULT NULL,
`location` varchar(128) NULL,
PRIMARY KEY (`item_id`),
KEY `item_type_id` (`item_type_id`),
KEY `brand_id` (`brand_id`),
KEY `site_id` (`site_id`),
KEY `seller_id` (`seller_id`)
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1;
I would also suggest using utf8_unicode_ci as the collate, utf8 as the charset, and InnoDB as the engine.
But first off, remove all those keys and try again. Also remove the aliasing on the 3rd query.
SELECT * FROM item WHERE brand_id = "10" AND item_type_id = "1" AND has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;
I have a query below that joins 4 tables. I have added indexes and the explain output looks good with indexes used (see below).
Can I optimize the query further?
The modelXml is rather big on some records.For a big project where I get back 22 records, each with approximately 2.5 - 3MB of modelXml data, the query is taking long (total of 69MB of data returned).
I suspect this is the issue but not sure how to deal with it.
I was reading on adjusting internal mysql variables e.g key_buffer_size and table_cache. Would any of this help?
key_buffer_size is currently set at 8384512 (~8MB) and table_cache at 64
What should I increase it to?
What other variables should I be looking at to manage to speed up return of such big data?
Any other suggestions are welcome. I am a novice to mysql but really trying to get better.
SELECT `m`.`modelId`, `m`.`modelTypeId`, `m`.`modelXml`, `m`.`xmlSize`, `m`.`createdById`, `m`.`creationDate`, `m`.`modifiedDate`, `u`.`firstName`, `u`.`lastName` FROM `models_1` AS `m`
INNER JOIN `modelFolderAssociations_1` AS `mfa` ON m.modelId = mfa.modelIOId
INNER JOIN `modelFolders_1` AS `mf` ON mfa.folderId = mf.folderId
INNER JOIN `users_1` AS `u` ON m.createdById = u.userId
WHERE (m.projectId = 2) AND (mfa.folderId = 5) AND (mfa.modelIOType = 2) AND (m.modelTypeId = 2)
CREATE TABLE `models` (
`modelId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`groupId` int(11) NOT NULL,
`projectId` int(11) NOT NULL,
`createdById` int(11) NOT NULL,
`modelTypeId` int(11) NOT NULL,
`modelXml` longtext,
`modelSpecXml` longtext NOT NULL,
`xmlSize` bigint(20) NOT NULL default '0',
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`modelId`,`customerId`)
) ENGINE=InnoDB AUTO_INCREMENT=75 DEFAULT CHARSET=utf8
CREATE TABLE `modelFolders` (
`folderId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`groupId` int(11) NOT NULL,
`projectId` int(11) NOT NULL,
`parentId` int(11) NOT NULL,
`folderName` varchar(64) NOT NULL,
`folderType` int(11) NOT NULL,
`editable` tinyint(1) NOT NULL default '1',
`nextDefaultNameNumber` int(11) NOT NULL default '1',
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`folderId`,`customerId`),
KEY `parentId` (`parentId`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=utf8
CREATE TABLE `modelFolderAssociations` (
`associationId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`folderId` int(11) NOT NULL,
`projectId` int(11) NOT NULL,
`modelIOId` int(11) NOT NULL,
`modelIOType` tinyint(1) NOT NULL default '1',
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`associationId`,`customerId`),
KEY `folderId` (`folderId`,`modelIOType`)
) ENGINE=InnoDB AUTO_INCREMENT=75 DEFAULT CHARSET=utf8
CREATE TABLE `users` (
`userId` int(11) NOT NULL auto_increment,
`customerId` int(11) NOT NULL,
`userName` varchar(50) NOT NULL,
`password` varchar(256) NOT NULL,
`firstName` varchar(50) default NULL,
`lastName` varchar(50) default NULL,
`creationDate` datetime NOT NULL,
`modifiedDate` datetime NOT NULL,
PRIMARY KEY (`userId`,`customerId`),
UNIQUE KEY `userName` (`userName`)
) ENGINE=InnoDB AUTO_INCREMENT=21 DEFAULT CHARSET=utf8
Explain output
+----+-------------+-------------------------+--------+---------------+----------+---------+---------------------------------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------------+--------+---------------+----------+---------+---------------------------------------------------+------+-------------+
| 1 | SIMPLE | modelFolders | const | PRIMARY | PRIMARY | 8 | const,const | 1 | Using index |
| 1 | SIMPLE | modelFolderAssociations | ref | folderId | folderId | 5 | const,const | 22 | Using where |
| 1 | SIMPLE | models | eq_ref | PRIMARY | PRIMARY | 8 | xa_system.modelFolderAssociations.modelIOId,const | 1 | Using where |
| 1 | SIMPLE | users | eq_ref | PRIMARY | PRIMARY | 8 | xa_system.models.createdById,const | 1 | |
+----+-------------+-------------------------+--------+---------------+----------+---------+---------------------------------------------------+------+-------------+
Having large text columns, like those you're using to store XML I presume, can hurt performance regardless of how well your indexes are structured.
It can be better in these cases to move the text columns to a separate table, indexed by string length which you're already storing and CRC32.
CREATE TABLE MODEL_XML (
xmlId INT(11) unsigned NOT NULL auto_increment,
xmlSize BIGINT(20) NOT NULL default '0',
crc32 INT(11) unsigned NOT NULL,
xmlData LONGTEXT,
PRIMARY KEY (xmlId),
UNIQUE KEY (xmlSize, crc32)
)
Then the width of your columns on the table with your important indices becomes constant.
ex.
modelXmlId INT(11) unsigned NOT NULL
specXmlId INT(11) unsigned NOT NULL
It also has the benefit of being more space-efficient for redundant text (empty strings, etc) since they would all share one xmlId and thus one row in the DB.
You should index your foreign keys, this will really help with your joins:
CREATE INDEX IDX_MODELS_CUSTID
on models (customerId)
CREATE INDEX IDX_FLDR_ASSOC_MODELIO
modelFolderAssociations(modelIOId)
CREATE INDEX IDX_FLDR_ASSOC_FLDRID
modelFolderAssociations(folderId)
and so on.
Test your query after taking off the modelXml column from the SELECT clause.
If the speed is then significantly better, then the slowness comes from the volume off data to transfer, not the query itself.
Your indexes don't always match the queries and joins. If all of the columns in your WHERE and JOIN clauses are not in the index, you're forcing mysql to look at the underlying row, which will kill your performance, especially since your model rows are so wide.
For "modelFolderAssociations", you made the correct composite key for the where clause, but you should include modelIOId for the join to model.
For "models", you'd want a composite index on (modelId, projectId, modelTypeId, createdById ) to cover the incoming link from mfa, the two items in the where clause, and the outbound link to users.
For "modelFolders" and "users", you have the incoming join covered in your primary keys.
The engine is only going to use one index, so adding extra index single indexes (as Mike suggested) won't be as good.