I have an Internal Inventory system with the below 3 tables as
a. Stocks - Daily updated from a CSV file.
---------------------------------
| id | MODELNO | Discount | MRP |
---------------------------------
| 1 | MODEL_1 | 40% | 900 |
| 2 | MODEL_A | 20% | 600 |
---------------------------------
Everyday this table is truncated and new stocks data are imported from a CSV file of a merchant.(around 6 Million records)
b. Cloths Master - The master clothes database
----------------------------------------
| ref_id | MODELNO | Name | MRP |
----------------------------------------
| 80 | MODEL_1 |Some Dress | 900 |
| 81 | MODEL_A |Another Dress| 600 |
----------------------------------------
The MODELNO is unique and ref_id the primary key. This table is part of the internal Inventory application (Has around 4.5 Million records)
c. Inventory table - It's part of the internal Applications
-------------------------------------------------
| id | ref_id | Name | MRP | status |
-------------------------------------------------
| 1 | 80 |Some Dress | 900 | ACTIVE |
| 2 | 81 |Another Dress| 600 | INACTIVE |
--------------------------------------------------
This table stores the available inventory for the product, based on the stocks and if the discount if above 40% the product is ACTIVE else by default INACTIVE.
The required functionality is that every day I need to run a script that would loop throught stock table records, and for the MODELNO update the stock on the Inventory table and If the record in Inventory table does not exist then it needs to be added.
What I have tried till now is a PHP script that would.
a. Firstly, set status in Inventory table for all records to INACTIVE.
b. And for each of the records in the stocks table, check if the MODELNO exists in Cloths Master table.
b. If the records exists then get the ref_id, and check if the ref_id exists in the Inventory Table and Update/Insert record accordingly.
The problem is that the script takes more than 8+ Hrs to complete.
Can you a suggest an efficient way, that can be used to implement the above functionality.
Note :
All the inserts and updated to the Inventory table are done using CodeIgniter's batch insert/update function.
I set all the status to INACTIVE, as there may be few products that are not present in the Stock DB.
The question in this case which comes in mind is - why not using a trigger ?
Create a table so_stocks
CREATE TABLE IF NOT EXISTS `so_stocks` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`MODELNO` varchar(50) COLLATE uft8_general_ci NOT NULL DEFAULT '0',
`Discount` int(10) DEFAULT '0',
`MRP` int(11) DEFAULT '0',
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=uft8_general_ci;
INSERT INTO `so_stocks` (`id`, `MODELNO`, `Discount`, `MRP`) VALUES
(1, 'MODEL_1', 40, 900),
(2, 'MODEL_A', 20, 600);
Create a table so_inventory
CREATE TABLE `so_inventory` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`ref_id` INT(11) NOT NULL DEFAULT '0',
`Name` VARCHAR(255) NOT NULL DEFAULT '0' COLLATE 'uft8_general_ci',
`MRP` INT(11) NOT NULL DEFAULT '0',
`status` TINYINT(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`)
)
COLLATE='uft8_general_ci'
ENGINE=MyISAM
AUTO_INCREMENT=1
;
And finally a table so_cloths
CREATE TABLE `so_cloths` (
`ref_id` INT(11) NOT NULL AUTO_INCREMENT,
`MODELNO` VARCHAR(50) NOT NULL DEFAULT '0' COLLATE 'uft8_general_ci',
`Name` VARCHAR(255) NOT NULL DEFAULT '0' COLLATE 'uft8_general_ci',
`MRP` INT(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`ref_id`)
)
COLLATE='uft8_general_ci'
ENGINE=MyISAM
AUTO_INCREMENT=1
;
And now the trigger
CREATE DEFINER=`root`#`::1` TRIGGER `so_cloths_after_insert` AFTER INSERT ON `so_cloths` FOR EACH ROW BEGIN
INSERT INTO so_inventory (ref_id,Name,MRP,status)
select sc.ref_id,sc.Name, sc.MRP, if (ss.Discount >= 40, 1,0) AS active from so_cloths AS sc
LEFT JOIN so_stocks AS ss ON (sc.MODELNO = ss.MODELNO)
WHERE sc.ref_id = new.ref_id;
END
Everytime you insert something into so_cloths an insert would be made into so_inventory.
Obviously it depends whether you want to insert data after inserting it into so_stocks or into so_cloths - you've to decide it - but the example should give you some insight.
The definer in the trigger statement has to be changed to your settings
I have a table with columns id,image and organisation name.I have inserted single image for each organisation initially but i need to update it with multiple images for each organisation.I am using php ver 5.4.16 and mysql ver 5.6.12.
Try to use a separator, for example use comma and store it like this in your table :
id image organization
1 path_to/img1.png org_name1
2 path_to/img1.png,path_to/img2.png org_name2
and later, after you extract the record, use explode function to extract it to an array like this :
$images = explode(",", $data->image);
PS : please give enough length for the image field, for example, give it varchar(4000), this is to make sure there will be no string truncation
Split the table in two and use the id of your table as foreign key in the new image table. ("normalization" and "relations" should be your search tags) https://en.wikipedia.org/wiki/Database_normalization
Or if you can't you should use json to insert multiple content. http://php.net/manual/en/book.json.php
You need to add a new table named "image" with columns :
- id_img
- image
- ref_organisation_id (foreign_key)
The best solution, in my opinion, to your problem would be to slightly redesign your database schema - the existing table will not be able to store multiple images for the same company judging by the overview of the tables you gave in the question.
There ought to be a table for orgainisations and another table for images associated with those organisations. The images table would have a key that links to the organisations table.
A very quickly put together example database structure
+----+------------------+
| id | name |
+----+------------------+
| 1 | ACME Web Company |
| 2 | ACME ISP |
+----+------------------+
+----+--------+------------+
| id | org_id | image |
+----+--------+------------+
| 1 | 1 | logo.jpg |
| 2 | 1 | banner.jpg |
| 3 | 1 | badge.png |
| 4 | 2 | logo.jpg |
| 5 | 2 | banner.gif |
+----+--------+------------+
create table if not exists `organisations` (
`id` int(10) unsigned not null auto_increment,
`name` varchar(50) not null,
primary key (`id`)
) engine=innodb auto_increment=3 default charset=utf8;
insert into `organisations` (`id`, `name`) values
(1, 'acme web company'),
(2, 'acme isp');
create table if not exists `org_images` (
`id` int(10) unsigned not null auto_increment,
`org_id` int(10) unsigned not null,
`image` varchar(50) not null,
primary key (`id`),
key `org_id` (`org_id`),
constraint `fk_org` foreign key (`org_id`) references `organisations` (`id`) on delete cascade on update cascade
) engine=innodb auto_increment=6 default charset=utf8;
insert into `org_images` (`id`, `org_id`, `image`) values
(1, 1, 'logo.jpg'),
(2, 1, 'banner.jpg'),
(3, 1, 'badge.png'),
(4, 2, 'logo.jpg'),
(5, 2, 'banner.gif');
Im creating a Rank and Requirement table for Martial Arts School.
Each student holds a rank in the martial arts. The rank name, belt color, and rank requirements are stored. Each rank will have numerous rank requirements. Each requirement is considered a requirement just for the rank at which the requirement is introduced. Every requirement is associated with a particular rank. All ranks except white belt have at least one requirement.
My ER Diagram:
Rank and Requirement ER Diagram
Rank Table:
CREATE TABLE IF NOT EXISTS `rank` (
`rank_id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`rank_nme` VARCHAR(45) NOT NULL,
PRIMARY KEY (`rank_id`))
ENGINE = InnoDB;
Output:
+------------+--------------+----------------+
| rank_id | INT(10) | AUTO_INCREMENT |
+------------+--------------+----------------+
| rank_nme | VARCHAR(45) | |
+------------+--------------+----------------+
Requirement Table:
CREATE TABLE IF NOT EXISTS `requirement` (
`req_id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`rank_id` INT UNSIGNED NOT NULL,
`req_nme` VARCHAR(45) NOT NULL,
`req_rank_nme` VARCHAR(45) NULL,
PRIMARY KEY (`req_id`),
INDEX `requirement_rank_id_idx` (`rank_id` ASC),
CONSTRAINT `requirement_rank_id_idx`
FOREIGN KEY (`rank_id`)
REFERENCES `rank` (`rank_id`)
ON DELETE RESTRICT
ON UPDATE CASCADE)
ENGINE = InnoDB;
Output:
+---------------+--------------+----------------+
| req_id | INT(10) | AUTO_INCREMENT |
+---------------+--------------+----------------+
| rank_id | INT(10) | |
+---------------+--------------+----------------+
| req_nme | VARCHAR(45) | |
+---------------+--------------+----------------+
| req_rank_nme | VARCHAR(45) | |
+---------------+--------------+----------------+
Need help if Im doing it right or wrong or you guys have modification or any suggestions! Thanks!
First in the first table you will need an id that is unique and auto increment for db good practice. Then you go with the rank_id, rank_name. If you want to associate the rank with a user, you will need another foreign key to do it. user_id from users.id .
Now you have rank_id which increments and i don't see the point of doing that. The rest is ok i think.
Table definition and queries explained:
item |
CREATE TABLE `item` (
`item_id` int(11) NOT NULL AUTO_INCREMENT,
`item_type_id` int(11) NOT NULL,
`brand_id` int(11) NOT NULL,
`site_id` int(11) NOT NULL,
`seller_id` int(11) NOT NULL,
`title` varchar(175) NOT NULL,
`desc` text NOT NULL,
`url` varchar(767) NOT NULL,
`price` int(11) NOT NULL,
`photo` varchar(255) NOT NULL,
`photo_file` varchar(255) NOT NULL,
`photo_type` varchar(32) NOT NULL,
`has_photo` enum('yes','no','pending') NOT NULL DEFAULT 'pending',
`added_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`updated_at` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`created_at` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`normalized_time` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`location` varchar(128) NOT NULL,
PRIMARY KEY (`item_id`),
KEY `item_type_id` (`item_type_id`),
KEY `brand_id` (`brand_id`),
KEY `site_id` (`site_id`),
KEY `seller_id` (`seller_id`),
KEY `created_at` (`created_at`),
KEY `added_at` (`added_at`),
KEY `normalized_time` (`normalized_time`),
KEY `typephototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `brandidphoto` (`brand_id`,`item_type_id`,`has_photo`),
KEY `brandidphoto2` (`brand_id`,`item_type_id`,`has_photo`),
KEY `idphoto` (`item_type_id`,`has_photo`),
KEY `idphototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `idphoto2` (`item_type_id`,`has_photo`),
KEY `typepricebrandid` (`item_type_id`,`price`,`brand_id`,`item_id`),
KEY `sellertypephototime` (`seller_id`,`item_type_id`,`has_photo`,`normalized_time`),
KEY `typephoto` (`item_type_id`,`has_photo`)
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1 |
mysql> explain SELECT item.* FROM item WHERE item.item_type_id = "1" AND item.has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
| 1 | SIMPLE | item | ref | item_type_id,typephototime,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | typephototime | 5 | const,const | 69528 | Using where |
+----+-------------+-------+------+------------------------------------------------------------------------------------+---------------+---------+-------------+-------+-------------+
1 row in set (0.02 sec)
mysql> explain SELECT * FROM item WHERE item_type_id = "1" AND (price BETWEEN "25" AND "275") AND brand_id = "10" ORDER BY item_id DESC LIMIT 1;
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | item | index | item_type_id,brand_id,typephototime,brandidphoto,brandidphoto2,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | PRIMARY | 4 | NULL | 203 | Using where |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+---------+---------+------+------+-------------+
1 row in set (0.01 sec)
mysql> explain SELECT item.* FROM item WHERE item.brand_id = "10" AND item.item_type_id = "1" AND item.has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
| 1 | SIMPLE | item | index | item_type_id,brand_id,typephototime,brandidphoto,brandidphoto2,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | normalized_time | 8 | NULL | 502397 | Using where |
+----+-------------+-------+-------+------------------------------------------------------------------------------------------------------------------------+-----------------+---------+------+--------+-------------+
1 row in set (2.15 sec)
mysql> explain SELECT COUNT(*) FROM item WHERE item.item_type_id = "1" AND item.has_photo = "yes" ;
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
| 1 | SIMPLE | item | ref | item_type_id,typephototime,idphoto,idphototime,idphoto2,typepricebrandid,typephoto | typephoto | 5 | const,const | 71135 | Using where; Using index |
+----+-------------+-------+------+------------------------------------------------------------------------------------+-----------+---------+-------------+-------+--------------------------+
1 row in set (0.01 sec)
The following indexes are redundant because they match the left columns of another index. You can almost certainly drop these indexes and save some space and overhead.
KEY `item_type_id` (`item_type_id`), /* redundant */
KEY `brand_id` (`brand_id`), /* redundant */
KEY `seller_id` (`seller_id`), /* redundant */
KEY `idphototime` (`item_type_id`,`has_photo`,`normalized_time`), /* redundant */
KEY `brandidphoto2` (`brand_id`,`item_type_id`,`has_photo`), /* redundant */
KEY `idphoto` (`item_type_id`,`has_photo`), /* redundant */
KEY `idphoto2` (`item_type_id`,`has_photo`), /* redundant */
KEY `typephoto` (`item_type_id`,`has_photo`) /* redundant */
That leaves the following indexes:
KEY `site_id` (`site_id`),
KEY `created_at` (`created_at`),
KEY `added_at` (`added_at`),
KEY `normalized_time` (`normalized_time`),
KEY `brandidphoto` (`brand_id`,`item_type_id`,`has_photo`),
KEY `typephototime` (`item_type_id`,`has_photo`,`normalized_time`),
KEY `typepricebrandid` (`item_type_id`,`price`,`brand_id`,`item_id`),
KEY `sellertypephototime` (`seller_id`,`item_type_id`,`has_photo`,`normalized_time`),
You can also use a tool like pt-duplicate-key-checker to find redundant indexes.
Next consider the storage engine:
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1;
Almost always, InnoDB is a better choice than MyISAM. Not only for performance, but for data integrity and crash safety. InnoDB has been the default storage engine since 2010, and it's the only storage engine that is actively getting improved. I'd recommend making a copy of this table, changing the storage engine to InnoDB, and compare its performance with respect to your queries.
Next let's consider indexes for the queries:
SELECT item.* FROM `item` WHERE item.item_type_id = "1" AND item.has_photo = "yes"
ORDER BY normalized_time DESC LIMIT 1;
I would choose an index on (item_type_id, has_photo, normalized_time) and that's the index it's currently using, which is typephototime.
One way to optimize this further would be to fetch only the columns in the index. That's when you see "Using index" in the EXPLAIN plan, it can be a huge improvement for performance.
Another important factor is to make sure that your index is cached in memory: increase key_buffer_size if you use MyISAM or innodb_buffer_pool_size if you use InnoDB to be as large as all the indexes you want to remain in memory. Because you don't want to run a query that needs to scan an index larger than your buffers; it causes a lot of swapping.
SELECT * FROM `item` WHERE item_type_id = "1" AND (price BETWEEN "25" AND "275") AND brand_id = "10"
ORDER BY item_id DESC LIMIT 1;
I would choose an index on (item_type_id, brand_id, price), but this query is currently using the PRIMARY index. You should create a new index.
SELECT item.* FROM `item` WHERE item.brand_id = "10" AND item.item_type_id = "1" AND item.has_photo = "yes"
ORDER BY normalized_time DESC LIMIT 1;
I would choose an index on (item_type_id, brand_id, has_photo, normalized_time). You should create a new index.
SELECT COUNT(*) FROM `item` WHERE item.item_type_id = "1" AND item.has_photo = "yes" ;
I would choose an index on (item_type_id, has_photo) and that's the index it's currently using, which is typephoto. It's also getting the "Using index" optimization, so the only other improvement could be to make sure there's enough buffer to hold the index in memory.
It's hard to optimize SELECT COUNT(*) queries because they naturally have to scan a lot of rows.
The other strategy to optimize COUNT(*) is to calculate the counts offline, and store them either in a summary table or in an in-memory cache like memcached so you don't have to recalculate them every time someone loads a page. But that means you have to update the counts every time someone adds or deletes a row in the item table, which could be more costly depending on how frequently that happens.
A few things I would suggest changing:
You don't need all those indexes. You really only need indexes on fields that are accessed a lot, like foreign key fields. Remove all the indexes except ones on ID fields.
You should be storing dates as nulls unless there is actual data.
Stay away from the enum data type, use smallint with flags representing each value. Example, 0 pending, 1 yes, 2 no.
Alongside reducing the size of the database, it makes things much cleaner. Your new table structure would look like so:
CREATE TABLE `item` (
`item_id` int(11) NOT NULL AUTO_INCREMENT,
`item_type_id` int(11) NOT NULL,
`brand_id` int(11) NOT NULL,
`site_id` int(11) NOT NULL,
`seller_id` int(11) NOT NULL,
`title` varchar(175) NOT NULL,
`desc` text NOT NULL,
`url` varchar(767) NOT NULL,
`price` int(11) NOT NULL,
`photo` varchar(255) NOT NULL,
`photo_file` varchar(255) NOT NULL,
`photo_type` varchar(32) NOT NULL,
`has_photo` smallint NOT NULL DEFAULT 0,
`added_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`updated_at` datetime NULL DEFAULT NULL,
`created_at` datetime NULL DEFAULT NULL,
`normalized_time` datetime NULL DEFAULT NULL,
`location` varchar(128) NULL,
PRIMARY KEY (`item_id`),
KEY `item_type_id` (`item_type_id`),
KEY `brand_id` (`brand_id`),
KEY `site_id` (`site_id`),
KEY `seller_id` (`seller_id`)
) ENGINE=MyISAM AUTO_INCREMENT=508885 DEFAULT CHARSET=latin1;
I would also suggest using utf8_unicode_ci as the collate, utf8 as the charset, and InnoDB as the engine.
But first off, remove all those keys and try again. Also remove the aliasing on the 3rd query.
SELECT * FROM item WHERE brand_id = "10" AND item_type_id = "1" AND has_photo = "yes" ORDER BY normalized_time DESC LIMIT 1;
I defined the column id as my primary key, but how do I make it automatically one larger than the last one?
You are lookin for AUTO_INCREMENT, you can check documentation here
You will need to set id column as AUTO_INCREMENT
Example from documentation
CREATE TABLE animals (
id MEDIUMINT NOT NULL AUTO_INCREMENT,
name CHAR(30) NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
you must set autoincrement.
CREATE TABLE animals (
id MEDIUMINT NOT NULL AUTO_INCREMENT,
name CHAR(30) NOT NULL,
PRIMARY KEY (id)
) ENGINE=MyISAM;
INSERT INTO animals (name) VALUES
('dog'),('cat'),('penguin'),
('lax'),('whale'),('ostrich');
SELECT * FROM animals;
Which returns:
+----+---------+
| id | name |
+----+---------+
| 1 | dog |
| 2 | cat |
| 3 | penguin |
| 4 | lax |
| 5 | whale |
| 6 | ostrich |
+----+---------+
http://dev.mysql.com/doc/refman/5.0/en/example-auto-increment.html mysql reference
Ok didn't notice the tags.
Hit the A_I checkbox in phpMyAdmin for the id column.
---old---
The Oracle way, triggered sequence:
CREATE sequence aic increment BY 1 start WITH 1;
CREATE TABLE blarg (
id NUMBER(15,0) PRIMARY KEY,
foobar VARCHAR2(255)
);
CREATE TRIGGER trigger ait BEFORE INSERT ON blarg
REFERENCING NEW AS NEW OLD AS OLD FOR EACH ROW
Begin
SELECT aic.NEXTVAL INTO :NEW.id FROM DUAL;
End;