Related
I have an Internal Inventory system with the below 3 tables as
a. Stocks - Daily updated from a CSV file.
---------------------------------
| id | MODELNO | Discount | MRP |
---------------------------------
| 1 | MODEL_1 | 40% | 900 |
| 2 | MODEL_A | 20% | 600 |
---------------------------------
Everyday this table is truncated and new stocks data are imported from a CSV file of a merchant.(around 6 Million records)
b. Cloths Master - The master clothes database
----------------------------------------
| ref_id | MODELNO | Name | MRP |
----------------------------------------
| 80 | MODEL_1 |Some Dress | 900 |
| 81 | MODEL_A |Another Dress| 600 |
----------------------------------------
The MODELNO is unique and ref_id the primary key. This table is part of the internal Inventory application (Has around 4.5 Million records)
c. Inventory table - It's part of the internal Applications
-------------------------------------------------
| id | ref_id | Name | MRP | status |
-------------------------------------------------
| 1 | 80 |Some Dress | 900 | ACTIVE |
| 2 | 81 |Another Dress| 600 | INACTIVE |
--------------------------------------------------
This table stores the available inventory for the product, based on the stocks and if the discount if above 40% the product is ACTIVE else by default INACTIVE.
The required functionality is that every day I need to run a script that would loop throught stock table records, and for the MODELNO update the stock on the Inventory table and If the record in Inventory table does not exist then it needs to be added.
What I have tried till now is a PHP script that would.
a. Firstly, set status in Inventory table for all records to INACTIVE.
b. And for each of the records in the stocks table, check if the MODELNO exists in Cloths Master table.
b. If the records exists then get the ref_id, and check if the ref_id exists in the Inventory Table and Update/Insert record accordingly.
The problem is that the script takes more than 8+ Hrs to complete.
Can you a suggest an efficient way, that can be used to implement the above functionality.
Note :
All the inserts and updated to the Inventory table are done using CodeIgniter's batch insert/update function.
I set all the status to INACTIVE, as there may be few products that are not present in the Stock DB.
The question in this case which comes in mind is - why not using a trigger ?
Create a table so_stocks
CREATE TABLE IF NOT EXISTS `so_stocks` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`MODELNO` varchar(50) COLLATE uft8_general_ci NOT NULL DEFAULT '0',
`Discount` int(10) DEFAULT '0',
`MRP` int(11) DEFAULT '0',
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=uft8_general_ci;
INSERT INTO `so_stocks` (`id`, `MODELNO`, `Discount`, `MRP`) VALUES
(1, 'MODEL_1', 40, 900),
(2, 'MODEL_A', 20, 600);
Create a table so_inventory
CREATE TABLE `so_inventory` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`ref_id` INT(11) NOT NULL DEFAULT '0',
`Name` VARCHAR(255) NOT NULL DEFAULT '0' COLLATE 'uft8_general_ci',
`MRP` INT(11) NOT NULL DEFAULT '0',
`status` TINYINT(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`)
)
COLLATE='uft8_general_ci'
ENGINE=MyISAM
AUTO_INCREMENT=1
;
And finally a table so_cloths
CREATE TABLE `so_cloths` (
`ref_id` INT(11) NOT NULL AUTO_INCREMENT,
`MODELNO` VARCHAR(50) NOT NULL DEFAULT '0' COLLATE 'uft8_general_ci',
`Name` VARCHAR(255) NOT NULL DEFAULT '0' COLLATE 'uft8_general_ci',
`MRP` INT(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`ref_id`)
)
COLLATE='uft8_general_ci'
ENGINE=MyISAM
AUTO_INCREMENT=1
;
And now the trigger
CREATE DEFINER=`root`#`::1` TRIGGER `so_cloths_after_insert` AFTER INSERT ON `so_cloths` FOR EACH ROW BEGIN
INSERT INTO so_inventory (ref_id,Name,MRP,status)
select sc.ref_id,sc.Name, sc.MRP, if (ss.Discount >= 40, 1,0) AS active from so_cloths AS sc
LEFT JOIN so_stocks AS ss ON (sc.MODELNO = ss.MODELNO)
WHERE sc.ref_id = new.ref_id;
END
Everytime you insert something into so_cloths an insert would be made into so_inventory.
Obviously it depends whether you want to insert data after inserting it into so_stocks or into so_cloths - you've to decide it - but the example should give you some insight.
The definer in the trigger statement has to be changed to your settings
Is there a way to INSERT a row into a table which contains an AUTO_INCREMENT column, and use the resulting AUTO_INCREMENT value to construct another field in the same row, in a single query?
I know it can be done in two operations.
Our product uses a 16 character numeric account ID stored as CHAR(16). We like to store the account ID as a single field as it is referenced in many tables throughout our database(s), however it is made up of the following parts:
AccountType INT(4) UNSIGNED // never less than 1000
AccountNumber INT(10) UNSIGNED AUTO_INCREMENT PRIMARY
AccountSite INT(2) UNSIGNED
Which are concatenated in order to produce the 16 character accountID field. The AccountType value is never less than 1000.
I would like to be able to perform a single INSERT operation that produces the four fields within the same query, if possible.
I can show this with a stored procedure. You may in the end only want 1 or 2 columns in the table. But I am showing more (assuming a text user friendly string too). That part is trivial, to shrink it down in column count.
Schema:
drop table if exists tX4;
create table tX4
( AccountType INT(4) UNSIGNED not null, -- // never less than 1000
AccountNumber INT(10) AUTO_INCREMENT PRIMARY key,
AccountSite INT(2) UNSIGNED not null,
col4 varchar(50) not null,
weWantThis char(16) not null
);
-- truncate tX4; -- used for testing
Stored Procedure:
DROP PROCEDURE IF EXISTS ins_tX4;
DELIMITER $$
CREATE PROCEDURE ins_tX4
( p_AccountType int,
p_AccountSite int,
p_col4 varchar(50)
)
BEGIN
DECLARE theAI int;
DECLARE sConcatFix char(16);
START TRANSACTION;
insert tX4(AccountType,AccountSite,col4,weWantThis) values (p_AccountType,p_AccountSite,p_col4,'');
set theAI=last_insert_id();
set sConcatFix=concat( lpad(p_AccountType,4,'0'), lpad(theAi,10,'0'), lpad(p_AccountSite,2,'0') );
update tX4 set weWantThis=sConcatFix where AccountNumber=theAI;
COMMIT;
END$$
DELIMITER ;
Test:
call ins_tX4(1,2,'cat');
select * from tX4;
+-------------+---------------+-------------+------+------------------+
| AccountType | AccountNumber | AccountSite | col4 | weWantThis |
+-------------+---------------+-------------+------+------------------+
| 1 | 1 | 2 | cat | 0001000000000102 |
+-------------+---------------+-------------+------+------------------+
Visualized pieces:
0001 0000000001 02
Which is AccountType, the AI, AccountSite (widths 4,10,2) respectively.
call ins_tX4(8765,42,'Sunday');
select * from tX4;
+-------------+---------------+-------------+--------+------------------+
| AccountType | AccountNumber | AccountSite | col4 | weWantThis |
+-------------+---------------+-------------+--------+------------------+
| 1 | 1 | 2 | cat | 0001000000000102 |
| 8765 | 2 | 42 | Sunday | 8765000000000242 |
+-------------+---------------+-------------+--------+------------------+
Column weWantThis could be used as the target for an FK from other tables.
Manual page for LPAD() (Left Padding).
Edit
With an OUT parameter to send back the AI # assigned. Note that parameters are IN parameters by default.
Stored Procedure:
DROP PROCEDURE IF EXISTS ins_tX4;
DELIMITER $$
CREATE PROCEDURE ins_tX4
( p_AccountType int,
p_AccountSite int,
p_col4 varchar(50),
OUT AI_assigned int
)
BEGIN
DECLARE theAI int;
DECLARE sConcatFix char(16);
-- AI means database-assigned AUTO INCREMENT
set AI_assigned = -1; -- assume an Error condition
START TRANSACTION;
insert tX4(AccountType,AccountSite,col4,weWantThis) values (p_AccountType,p_AccountSite,p_col4,'');
set theAI=last_insert_id();
set sConcatFix=concat( lpad(p_AccountType,4,'0'), lpad(theAi,10,'0'), lpad(p_AccountSite,2,'0') );
update tX4 set weWantThis=sConcatFix where AccountNumber=theAI;
set AI_assigned = theAI; -- the OUT parameter is set to the AI value
COMMIT;
END$$
DELIMITER ;
Test from mysql environment:
set #var1 = -1;
call ins_tX4(22,33,'Monday',#var1);
select #var1;
-- 3
call ins_tX4(333,79,'Tuesday',#var1);
select #var1;
-- 4
Test from a PHP environment:
Because this question is tagged PHP, see an answer like the one here from user Matteo Tassinari.
This can be accomplished with a TRANSACTION.
START TRANSACTION;
INSERT INTO accounts (accountType, accountSite) VALUES(1000,10);
UPDATE accounts SET accountID = CONCAT(LPAD(accountType, 4, '0'), LPAD(accountNumber, 10, '0'), LPAD(accountSite, 2, '0')) WHERE accountNumber = LAST_INSERT_ID();
COMMIT;
Which could for example be executed in php as:
$mysqli = new mysqli(...);
$mysqli->begin_transaction();
$query = "INSERT INTO accounts (accountType, accountSite) VALUES(1000,10);"
$query .= "UPDATE accounts SET accountID = CONCAT(LPAD(accountType, 4, '0'), LPAD(accountNumber, 10, '0'), LPAD(accountSite, 2, '0')) WHERE accountNumber = LAST_INSERT_ID();"
$mysqli->query($query);
$mysqli->commit();
My MySQL app allows users to create bookmarks and organize them with tags.
A tag is added to a bookmark with a middle MySQL DB table bookmark_tag_relationship
I would like to add a Foreign Key to auto-delete all records in bookmark_tag_relationship when a Bookmarks record is deleted.
All the bookmark_tag_relationship records deleted would have a bookmark_id column that matches the id column on the bookmark record that is deleted
How can I set this up correctly based on the 3 table structures shown below?
Failed attempt #1
I tried something like this:
ALTER TABLE bookmarks ADD FOREIGN KEY (`id`)
REFERENCES bookmark_tag_relationship(`bookmark_id`);
It returned an error with no information just that it could not create it.
table bookmarks
--
-- Table structure for table `bookmarks`
--
CREATE TABLE IF NOT EXISTS `bookmarks` (
`id` int(20) NOT NULL,
`url_id` int(20) DEFAULT NULL,
`user_id` int(30) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`url` varchar(255) DEFAULT NULL,
`favicon_url` varchar(255) DEFAULT NULL,
`project_url` varchar(255) DEFAULT NULL,
`github_url` varchar(255) DEFAULT NULL,
`demo_url` int(255) DEFAULT NULL,
`local_demo_url` varchar(255) DEFAULT NULL,
`image1` varchar(255) DEFAULT NULL,
`image2` varchar(255) DEFAULT NULL,
`image3` varchar(255) DEFAULT NULL,
`image4` varchar(255) DEFAULT NULL,
`description` text,
`notes` text,
`tags_string` varchar(255) DEFAULT NULL,
`click_count` int(10) NOT NULL DEFAULT '0',
`tag_count` int(10) NOT NULL DEFAULT '0',
`created_on` datetime DEFAULT NULL,
`updated_on` datetime DEFAULT NULL,
`last_viewed_on` datetime DEFAULT NULL,
`active` int(1) NOT NULL DEFAULT '1'
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
table tags
--
-- Table structure for table `tags`
--
CREATE TABLE IF NOT EXISTS `tags` (
`id` int(11) NOT NULL,
`user_id` int(11) DEFAULT NULL,
`name` varchar(100) DEFAULT NULL,
`description` text,
`color` varchar(10) DEFAULT NULL,
`bookmark_count` int(11) NOT NULL DEFAULT '0',
`active` int(11) NOT NULL DEFAULT '1'
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latin1;
table bookmark_tag_relationship
--
-- Table structure for table `bookmark_tag_relationship`
--
CREATE TABLE IF NOT EXISTS `bookmark_tag_relationship` (
`id` int(11) NOT NULL,
`bookmark_id` int(30) NOT NULL,
`tag_id` int(30) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Three issues:
You have not defined a primary key in the table bookmarks (or any other);
In comments you clarified you actually have them, so then this is a non-issue;
You have the foreign key constraint defined in the opposite direction. You should define it on the child table;
You need to add the ON DELETE CASCADE clause.
So, execute this, and it will work (you can skip the first one, since in comments you clarified you already have the primary key):
ALTER TABLE bookmarks ADD
CONSTRAINT PRIMARY KEY(id);
ALTER TABLE bookmark_tag_relationship
ADD FOREIGN KEY (bookmark_id)
REFERENCES bookmarks(id)
ON DELETE CASCADE;
Now you can delete a bookmark, for example:
DELETE FROM bookmarks WHERE id = 1;
...and the related records in bookmark_tag_relationship will be deleted in the same transaction.
Some additional remarks:
The difference between int(20) and int(30) that you have for the primary key and foreign key types cannot be very useful. It does not influence the constraint, but I would still suggest to harmonise that as follows:
ALTER TABLE bookmark_tag_relationship
MODIFY bookmark_id int(20) NOT NULL;
It is not needed to surround your table and column names with backticks. Only when you have chosen reserved words for those names, they are needed, but that is something you'd want to avoid anyway.
EDIT: You're foreign key is also failing because you do not have a primary key. A foreign key is a reference to a primary key.
I would setup a trigger for bookmarks on delete.
DELIMITER $$
CREATE DEFINER=`username`#`%` TRIGGER `bookmarks_ADEL` AFTER DELETE ON `bookmarks` FOR EACH ROW
BEGIN
DELETE FROM bookmark_tag_relationship WHERE bookmark_id = old.id;
END
When a row from Bookmarks is deleted, it will take that ID and remove the row out of bookmark_tag_relationship.
You need to set Primary keys in both those tables, most likely the ID
One note about using a cascade delete on foreign key. It will not activate any triggers that are set on those deleted rows. So as you progress, if you decide to trigger off them to remove or modify more information, the foreign key cascade will not activate them and you'll have to trigger on delete anyway.
Try to add a ON DELETE CASCADE on your foreign key.
ALTER TABLE bookmark_tag_relationship ADD FOREIGN KEY (`bookmark_id`)
REFERENCES bookmarks(`id`)
ON DELETE CASCADE;
There's a few reasons why your foreign key is not working.
First of all, you need to make sure the column id of each table is a primary key.
ALTER TABLE table_name
ADD PRIMARY KEY (table_column)
After you've added your primary key, you need to make sure that their type is the same.
In your bookmark_tag_relationship table, the bookmark_id is INT(30), but in your bookmarks* table, your column 'id' is **INT(20). You need to make sure both of them are the same. So either change the id from bookmarks to INT(30) or change the other one to INT(20).
And at last, your foreign key query is reversed. You want it the other way around.
ALTER TABLE bookmark_tag_relationship ADD FOREIGN KEY (`bookmark_id`)
REFERENCES bookmarks(`id`)
ON DELETE CASCADE;
You want the bookmark_tag_relationship to be deleted when a bookmark is deleted, and not the other way around.
Use a primary auto incremented key on the id columns. And then like others said use cascade on delete with foreign keys.
mysql> show create table bookmarks\G
*************************** 1. row ***************************
Table: bookmarks
Create Table: CREATE TABLE `bookmarks` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
mysql> show create table tags\G
*************************** 1. row ***************************
Table: tags
Create Table: CREATE TABLE `tags` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
mysql> show create table bookmarks_tags\G
*************************** 1. row ***************************
Table: bookmarks_tags
Create Table: CREATE TABLE `bookmarks_tags` (
`bookmark_id` int(11) NOT NULL DEFAULT '0',
`tag_id` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`bookmark_id`,`tag_id`),
CONSTRAINT `bookmarks_tags_ibfk_1` FOREIGN KEY (`bookmark_id`) REFERENCES `bookmarks` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.01 sec)
mysql> select * from bookmarks;
+----+-----------+
| id | title |
+----+-----------+
| 1 | bookmark1 |
| 2 | bookmark2 |
| 3 | bookmark3 |
+----+-----------+
3 rows in set (0.00 sec)
mysql> select * from tags;
+----+------+
| id | name |
+----+------+
| 1 | tag1 |
| 2 | tag2 |
| 3 | tag3 |
| 4 | tag4 |
| 5 | tag5 |
+----+------+
5 rows in set (0.00 sec)
mysql> select * from bookmarks_tags;
+-------------+--------+
| bookmark_id | tag_id |
+-------------+--------+
| 1 | 1 |
| 1 | 2 |
| 1 | 3 |
| 2 | 1 |
| 2 | 4 |
| 3 | 5 |
+-------------+--------+
6 rows in set (0.00 sec)
mysql> delete from bookmarks where title = 'bookmark1';
Query OK, 1 row affected (0.01 sec)
mysql> select * from bookmarks; select * from tags; select * from bookmarks_tags;
+----+-----------+
| id | title |
+----+-----------+
| 2 | bookmark2 |
| 3 | bookmark3 |
+----+-----------+
2 rows in set (0.00 sec)
+----+------+
| id | name |
+----+------+
| 1 | tag1 |
| 2 | tag2 |
| 3 | tag3 |
| 4 | tag4 |
| 5 | tag5 |
+----+------+
5 rows in set (0.01 sec)
+-------------+--------+
| bookmark_id | tag_id |
+-------------+--------+
| 2 | 1 |
| 2 | 4 |
| 3 | 5 |
+-------------+--------+
3 rows in set (0.00 sec)
I have a table with columns id,image and organisation name.I have inserted single image for each organisation initially but i need to update it with multiple images for each organisation.I am using php ver 5.4.16 and mysql ver 5.6.12.
Try to use a separator, for example use comma and store it like this in your table :
id image organization
1 path_to/img1.png org_name1
2 path_to/img1.png,path_to/img2.png org_name2
and later, after you extract the record, use explode function to extract it to an array like this :
$images = explode(",", $data->image);
PS : please give enough length for the image field, for example, give it varchar(4000), this is to make sure there will be no string truncation
Split the table in two and use the id of your table as foreign key in the new image table. ("normalization" and "relations" should be your search tags) https://en.wikipedia.org/wiki/Database_normalization
Or if you can't you should use json to insert multiple content. http://php.net/manual/en/book.json.php
You need to add a new table named "image" with columns :
- id_img
- image
- ref_organisation_id (foreign_key)
The best solution, in my opinion, to your problem would be to slightly redesign your database schema - the existing table will not be able to store multiple images for the same company judging by the overview of the tables you gave in the question.
There ought to be a table for orgainisations and another table for images associated with those organisations. The images table would have a key that links to the organisations table.
A very quickly put together example database structure
+----+------------------+
| id | name |
+----+------------------+
| 1 | ACME Web Company |
| 2 | ACME ISP |
+----+------------------+
+----+--------+------------+
| id | org_id | image |
+----+--------+------------+
| 1 | 1 | logo.jpg |
| 2 | 1 | banner.jpg |
| 3 | 1 | badge.png |
| 4 | 2 | logo.jpg |
| 5 | 2 | banner.gif |
+----+--------+------------+
create table if not exists `organisations` (
`id` int(10) unsigned not null auto_increment,
`name` varchar(50) not null,
primary key (`id`)
) engine=innodb auto_increment=3 default charset=utf8;
insert into `organisations` (`id`, `name`) values
(1, 'acme web company'),
(2, 'acme isp');
create table if not exists `org_images` (
`id` int(10) unsigned not null auto_increment,
`org_id` int(10) unsigned not null,
`image` varchar(50) not null,
primary key (`id`),
key `org_id` (`org_id`),
constraint `fk_org` foreign key (`org_id`) references `organisations` (`id`) on delete cascade on update cascade
) engine=innodb auto_increment=6 default charset=utf8;
insert into `org_images` (`id`, `org_id`, `image`) values
(1, 1, 'logo.jpg'),
(2, 1, 'banner.jpg'),
(3, 1, 'badge.png'),
(4, 2, 'logo.jpg'),
(5, 2, 'banner.gif');
Table definition and queries explained:
item |
CREATE TABLE `item` (
`item_id` int(11) NOT NULL AUTO_INCREMENT,
`item_type_id` int(11) NOT NULL,
`brand_id` int(11) NOT NULL,
`site_id` int(11) NOT NULL,
`seller_id` int(11) NOT NULL,
`title` varchar(175) NOT NULL,
`desc` text NOT NULL,
`url` varchar(767) NOT NULL,
`price` int(11) NOT NULL,
`photo` varchar(255) NOT NULL,
`photo_file` varchar(255) NOT NULL,
`photo_type` varchar(32) NOT NULL,
`has_photo` enum('yes','no','pending') NOT NULL DEFAULT 'pending',
`added_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`updated_at` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`created_at` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`normalized_time` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`location` varchar(128) NOT NULL,
PRIMARY KEY (`item_id`),
KEY `item_type_id` (`item_type_id`),
KEY `brand_id` (`brand_id`),
KEY `site_id` (`site_id`),
KEY `seller_id` (`seller_id`),
KEY `created_at` (`created_at`),
KEY `added_at` (`added_at`),
KEY `normalized_time` (`normalized_time`),
KEY `typephototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `brandidphoto` (`brand_id`,`item_type_id`,`has_photo`),
KEY `brandidphoto2` (`brand_id`,`item_type_id`,`has_photo`),
KEY `idphoto` (`item_type_id`,`has_photo`),
KEY `idphototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `idphoto2` (`item_type_id`,`has_photo`),
KEY `typepricebrandid` (`item_type_id`,`price`,`brand_id`,`item_id`),
KEY `sellertypephototime` (`seller_id`,`item_type_id`,`has_photo`,`normalized_time`),
KEY `typephoto` (`item_type_id`,`has_photo`)
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1 |
mysql> explain SELECT item.* FROM item WHERE item.item_type_id = "1" AND item.has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
| 1 | SIMPLE | item | ref | item_type_id,typephototime,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | typephototime | 5 | const,const | 69528 | Using where |
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
1 row in set (0.02 sec)
mysql> explain SELECT * FROM item WHERE item_type_id = "1" AND (price BETWEEN "25" AND "275") AND brand_id = "10" ORDER BY item_id DESC LIMIT 1;
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | item | index | item_type_id,brand_id,typephototime,brandidphoto,brandidphoto2,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | PRIMARY | 4 | NULL | 203 | Using where |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
1 row in set (0.01 sec)
mysql> explain SELECT item.* FROM item WHERE item.brand_id = "10" AND item.item_type_id = "1" AND item.has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
| 1 | SIMPLE | item | index | item_type_id,brand_id,typephototime,brandidphoto,brandidphoto2,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | normalized_time | 8 | NULL | 502397 | Using where |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
1 row in set (2.15 sec)
mysql> explain SELECT COUNT(*) FROM item WHERE item.item_type_id = "1" AND item.has_photo = "yes" ;
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
| 1 | SIMPLE | item | ref | item_type_id,typephototime,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | typephoto | 5 | const,const | 71135 | Using where; Using index |
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
1 row in set (0.01 sec)
The following indexes are redundant because they match the left columns of another index. You can almost certainly drop these indexes and save some space and overhead.
KEY `item_type_id` (`item_type_id`), /* redundant */
KEY `brand_id` (`brand_id`), /* redundant */
KEY `seller_id` (`seller_id`), /* redundant */
KEY `idphototime` (`item_type_id`,`has_photo`,`normalized_time`), /* redundant */
KEY `brandidphoto2` (`brand_id`,`item_type_id`,`has_photo`), /* redundant */
KEY `idphoto` (`item_type_id`,`has_photo`), /* redundant */
KEY `idphoto2` (`item_type_id`,`has_photo`), /* redundant */
KEY `typephoto` (`item_type_id`,`has_photo`) /* redundant */
That leaves the following indexes:
KEY `site_id` (`site_id`),
KEY `created_at` (`created_at`),
KEY `added_at` (`added_at`),
KEY `normalized_time` (`normalized_time`),
KEY `brandidphoto` (`brand_id`,`item_type_id`,`has_photo`),
KEY `typephototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `typepricebrandid` (`item_type_id`,`price`,`brand_id`,`item_id`),
KEY `sellertypephototime` (`seller_id`,`item_type_id`,`has_photo`,`normalized_time`),
You can also use a tool like pt-duplicate-key-checker to find redundant indexes.
Next consider the storage engine:
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1;
Almost always, InnoDB is a better choice than MyISAM. Not only for performance, but for data integrity and crash safety. InnoDB has been the default storage engine since 2010, and it's the only storage engine that is actively getting improved. I'd recommend making a copy of this table, changing the storage engine to InnoDB, and compare its performance with respect to your queries.
Next let's consider indexes for the queries:
SELECT item.* FROM `item` WHERE item.item_type_id = "1" AND item.has_photo = "yes"
ORDER BY normalized_time DESC LIMIT 1;
I would choose an index on (item_type_id, has_photo, normalized_time) and that's the index it's currently using, which is typephototime.
One way to optimize this further would be to fetch only the columns in the index. That's when you see "Using index" in the EXPLAIN plan, it can be a huge improvement for performance.
Another important factor is to make sure that your index is cached in memory: increase key_buffer_size if you use MyISAM or innodb_buffer_pool_size if you use InnoDB to be as large as all the indexes you want to remain in memory. Because you don't want to run a query that needs to scan an index larger than your buffers; it causes a lot of swapping.
SELECT * FROM `item` WHERE item_type_id = "1" AND (price BETWEEN "25" AND "275") AND brand_id = "10"
ORDER BY item_id DESC LIMIT 1;
I would choose an index on (item_type_id, brand_id, price), but this query is currently using the PRIMARY index. You should create a new index.
SELECT item.* FROM `item` WHERE item.brand_id = "10" AND item.item_type_id = "1" AND item.has_photo = "yes"
ORDER BY normalized_time DESC LIMIT 1;
I would choose an index on (item_type_id, brand_id, has_photo, normalized_time). You should create a new index.
SELECT COUNT(*) FROM `item` WHERE item.item_type_id = "1" AND item.has_photo = "yes" ;
I would choose an index on (item_type_id, has_photo) and that's the index it's currently using, which is typephoto. It's also getting the "Using index" optimization, so the only other improvement could be to make sure there's enough buffer to hold the index in memory.
It's hard to optimize SELECT COUNT(*) queries because they naturally have to scan a lot of rows.
The other strategy to optimize COUNT(*) is to calculate the counts offline, and store them either in a summary table or in an in-memory cache like memcached so you don't have to recalculate them every time someone loads a page. But that means you have to update the counts every time someone adds or deletes a row in the item table, which could be more costly depending on how frequently that happens.
A few things I would suggest changing:
You don't need all those indexes. You really only need indexes on fields that are accessed a lot, like foreign key fields. Remove all the indexes except ones on ID fields.
You should be storing dates as nulls unless there is actual data.
Stay away from the enum data type, use smallint with flags representing each value. Example, 0 pending, 1 yes, 2 no.
Alongside reducing the size of the database, it makes things much cleaner. Your new table structure would look like so:
CREATE TABLE `item` (
`item_id` int(11) NOT NULL AUTO_INCREMENT,
`item_type_id` int(11) NOT NULL,
`brand_id` int(11) NOT NULL,
`site_id` int(11) NOT NULL,
`seller_id` int(11) NOT NULL,
`title` varchar(175) NOT NULL,
`desc` text NOT NULL,
`url` varchar(767) NOT NULL,
`price` int(11) NOT NULL,
`photo` varchar(255) NOT NULL,
`photo_file` varchar(255) NOT NULL,
`photo_type` varchar(32) NOT NULL,
`has_photo` smallint NOT NULL DEFAULT 0,
`added_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`updated_at` datetime NULL DEFAULT NULL,
`created_at` datetime NULL DEFAULT NULL,
`normalized_time` datetime NULL DEFAULT NULL,
`location` varchar(128) NULL,
PRIMARY KEY (`item_id`),
KEY `item_type_id` (`item_type_id`),
KEY `brand_id` (`brand_id`),
KEY `site_id` (`site_id`),
KEY `seller_id` (`seller_id`)
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1;
I would also suggest using utf8_unicode_ci as the collate, utf8 as the charset, and InnoDB as the engine.
But first off, remove all those keys and try again. Also remove the aliasing on the 3rd query.
SELECT * FROM item WHERE brand_id = "10" AND item_type_id = "1" AND has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;