MySQL: Key/value store queries optimization - php

I am seeking a help with setting up correct indexes (i've tried too many and now i am a bit lost), correct MySQL engine (MyIsam, InnoDB...) and help with my queries (JOINs, ...). Also I have headache when I am thinking that these queries should return count(*).
My times were more than 5 - 10 seconds per query without counting, but I am not sure if I can get better times for this big database.
I am trying to optimize this MySQL tables:
Items (~600k rows):
+-------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| type | varchar(255) | NO | PRI | NULL | |
+-------+------------------+------+-----+---------+----------------+
Items_Relationships (~1M rows):
+-------------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------+-------+
| lft_item_id | int(11) unsigned | NO | PRI | NULL | |
| rgt_item_id | int(11) unsigned | NO | PRI | NULL | |
| rel_type | varchar(255) | NO | PRI | NULL | |
+-------------+------------------+------+-----+---------+-------+
Items_Values (~4M rows):
+---------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| item_id | int(11) unsigned | NO | PRI | 0 | |
| name | varchar(255) | YES | MUL | NULL | |
| value | longtext | YES | | NULL | |
| lang | varchar(2) | YES | | NULL | |
+---------+------------------+------+-----+---------+----------------+
I am running basically these common queries:
1. query - All items with value "status" > 1:
SELECT `company`.`id` AS `id`,
`company`.`type` AS `type`
FROM `items` AS `company`
INNER JOIN `items_values` AS `value_name` ON (`company`.`id` = `value_name`.`item_id`)
WHERE `company`.`type` = 'company'
AND `value_name`.`name` = 'status'
AND CONVERT(`value_name`.`value`, SIGNED) > 1
GROUP BY `company`.`id`
ORDER BY `company`.`id` DESC
LIMIT 0, 30
2. query - All items with some values in relationship with other items:
SELECT `company`.`id` AS `id`,
`company`.`type` AS `type`
FROM `items` AS `company`
INNER JOIN `items_values` AS `value_status` ON (`value_status`.`item_id` = `company`.`id`)
INNER JOIN `items_relationships` AS `companies_categories` ON (`companies_categories`.`lft_item_id` = `company`.`id`)
INNER JOIN `items_values` AS `category_rgt` ON (`category_rgt`.`item_id` = `companies_categories`.`rgt_item_id`)
WHERE `company`.`type` = 'company'
AND `company`.`type` = 'company'
AND `value_status`.`name` = 'status'
AND CONVERT(`value_status`.`value`, SIGNED) >= 1
AND `category_rgt`.`name` = 'rgt'
AND (CONVERT(category_rgt.value, UNSIGNED) BETWEEN 2805 AND 4222)
AND `companies_categories`.`rel_type` = 'company_category'
GROUP BY `company`.`id`
ORDER BY `company`.`id` DESC LIMIT 10
OFFSET 0
Thx in advance!

If you ask about indexes, so you have almost everything you need already indexed.
I have just question about
| name | varchar(255) | YES | MUL | NULL | |
So I would prefer to set it as well as primary key.
I have only one suggestion about table structures.
If you have mixed string and numbers in your
| value | longtext | YES | | NULL | |
create another column int_value SIGNED or unsigned is even better.
And you should set that column as index as well (as soon as you need that column used as a filter and/or search criteria)
And fill that field on insert/update where it is applicable.
This modification will increase performance of query where you should not use CAST and/or CONVERT for millions of records like you do here:
AND CONVERT(`value_status`.`value`, SIGNED) >= 1
AND (CONVERT(category_rgt.value, UNSIGNED) BETWEEN 2805 AND 4222)
So I have no more comments about structure.
But I would ask you to try my query, just like an experiment if it faster than yours. Unfortunately I have no possibility to debug with any data. If you provide some sqlfiddle that will help a lot.
SELECT `company`.`id` AS `id`,
`company`.`type` AS `type`
FROM `items` AS `company`
INNER JOIN (
SELECT
item_id,
FROM items_values
WHERE name = 'status'
AND CONVERT(value, SIGNED) >= 1
) AS value_status
ON value_status.item_id = company.id
INNER JOIN
(
SELECT
lft_item_id
FROM
items_relationships
INNER JOIN (
SELECT
item_id
FROM
items_values
WHERE name = 'rgt'
AND (CONVERT(value, UNSIGNED) BETWEEN 2805 AND 4222)
) AS category_rgt
ON category_rgt.item_id = items_relationships.rgt_item_id
WHERE items_relationships.rel_type = 'company_category'
) as companies_categories
ON (`companies_categories`.`lft_item_id` = `company`.`id`)
WHERE `company`.`type` = 'company'
GROUP BY `company`.`id`
ORDER BY `company`.`id` DESC
LIMIT 10

Related

Insert trigger subtract value from another table

I am a newbie at sql. I made a php website that user can make item purchases. And now I want to make a trigger for when a receipt is created, it subtracts the quantity in the item stock table with the quantity sold in the receipt.
create table if not exists Store.Receipt (
ReceiptID varchar(10) NOT NULL DEFAULT '0',
ItemID varchar(10) NOT NULL,
PriceSold int NOT NULL,
QtySold int NOT NULL,
RDate DATETIME NOT NULL,
BranchID varchar(10) NOT NULL,
EmployeeID varchar(10) NOT NULL,
MemberID varchar(10) DEFAULT "NotMember",
primary key (ReceiptID, ItemID));
alter table Store.Receipt
add constraint FK_Receipt_Member foreign key (MemberID)
references Store.MemberShip (MemberID),
add constraint FK_Receipt_Employee foreign key (EmployeeID)
references Store.Employee (EmployeeID),
add constraint FK_Receipt_Branch foreign key (BranchID)
references Store.Branch (BranchID)
on delete restrict
on update cascade;
The sql script above is for the receipt table. The id for this table is auto generated using autoincrement. One ReceiptID can contain many items. So I am using a composite of ReceiptID and ItemID sold as the primary key. And here is the Script for the table.
create table if not exists Store.Item (
ItemID varchar(10) NOT NULL DEFAULT '0',
ItemName varchar(45) NOT NULL,
UnitPrice int NOT NULL,
QtyStock int NOT NULL,
SupplierID varchar(10) NOT NULL,
primary key (ItemID));
alter table Store.Item
add constraint FK_Supplier foreign key (SupplierID)
references Store.Supplier (SupplierID)
on delete restrict
on update cascade;
I have been trying to make the trigger on my own, but I am still quite unfimiliar with sql triggers. Here is what I came up with.
DELIMITER $$
CREATE TRIGGER update_stock_trigger
AFTER INSERT ON Store.Receipt
FOR EACH ROW
BEGIN
DECLARE
#qty numeric
#itemid varchar
#itemid = SELECT ItemID FROM Receipt WHERE ReceiptID = INSERTED.ItemID;
#qty = SELECT QtySold FROM Receipt WHERE ReceiptID = INSERTED.QtySold;
UPDATE Item SET QtyStock = QtyStock - #qty WHERE ItemID = #itemid;
END$$
DELIMITER ;
Thank you for your help in advance.
I don't think the trigger needs to be quite so complicated as you can access the values required directly by prefixing the column name with NEW
create trigger `update_stock_trigger`
after insert on `receipt`
for each row
begin
update `item` set `qtystock` = `qtystock` - new.qtysold where `itemid` = new.itemid;
end
Using a slightly simplified schema ( remove FK dependency upon Supplier table )
mysql> describe item;
+------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+-------+
| ItemID | varchar(10) | NO | PRI | 0 | |
| ItemName | varchar(45) | NO | | NULL | |
| UnitPrice | int(11) | NO | | NULL | |
| QtyStock | int(11) | NO | | NULL | |
| SupplierID | varchar(10) | NO | | NULL | |
+------------+-------------+------+-----+---------+-------+
mysql> describe receipt;
+------------+-------------+------+-----+-----------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+-----------+-------+
| ReceiptID | varchar(10) | NO | PRI | 0 | |
| ItemID | varchar(10) | NO | PRI | NULL | |
| PriceSold | int(11) | NO | | NULL | |
| QtySold | int(11) | NO | | NULL | |
| RDate | datetime | NO | | NULL | |
| BranchID | varchar(10) | NO | | NULL | |
| EmployeeID | varchar(10) | NO | | NULL | |
| MemberID | varchar(10) | YES | | NotMember | |
+------------+-------------+------+-----+-----------+-------+
And populated a single Product in the item table
mysql> select * from item;
+--------+----------+-----------+----------+------------+
| ItemID | ItemName | UnitPrice | QtyStock | SupplierID |
+--------+----------+-----------+----------+------------+
| 1 | bob | 23 | 100 | 404-blah |
+--------+----------+-----------+----------+------------+
And an initial record in the receipt table
mysql> select * from receipt;
+-----------+--------+-----------+---------+---------------------+----------+------------+-----------+
| ReceiptID | ItemID | PriceSold | QtySold | RDate | BranchID | EmployeeID | MemberID |
+-----------+--------+-----------+---------+---------------------+----------+------------+-----------+
| 1 | 1 | 43 | 17 | 2020-01-30 09:28:29 | Dundee | Senga-X | NotMember |
+-----------+--------+-----------+---------+---------------------+----------+------------+-----------+
and a subsequent second record ( showing used sql )
INSERT INTO `Store`.`receipt` (`ReceiptID`, `ItemID`, `PriceSold`, `QtySold`, `RDate`, `BranchID`, `EmployeeID`)
VALUES ('2', '1', '35', '10', '2020-01-30 09:32:00', 'Glasgow', 'Senga-Y');
The trigger has updated the item table accordingly
mysql> select * from item;
+--------+----------+-----------+----------+------------+
| ItemID | ItemName | UnitPrice | QtyStock | SupplierID |
+--------+----------+-----------+----------+------------+
| 1 | bob | 23 | 73 | 404-blah |
+--------+----------+-----------+----------+------------+

mysql query - not scaling well, slow with 15k records

I have a large SQL query (mysql) used to populate a table of records in a CRM system.
This has been working well and quite fast with around 4000 records. Now reaching 15,500 it's running painfully slow. Taking around 70 seconds to return data.
I have tried adding some indexes, but with limited success. Any suggestions?
The query is:
SELECT SQL_NO_CACHE SQL_CALC_FOUND_ROWS
s.*,
ca2.address_postcode,
ca2.resident_status
FROM
(
SELECT
a.id,
c.name_first,
c.name_last,
a.created as app_created,
a.edited,
a.comments as custcomment,
c.name_company,
a.loan_amount,
a.product_type,
ap.packager as placed_with,
a.introducer_id,
a.status,
c.contact_number,
c.email,
a.database,
bd.legal_structure,
(
SELECT
ca1.id
FROM
cl_customer_address AS ca1
WHERE
ca1.customer_id = a.customer_id AND
ca1.deleted = "0000-00-00 00:00:00"
ORDER BY
ca1.created DESC
LIMIT
1
) AS address_id
FROM
cl_application AS a
LEFT JOIN
cl_application_business_details as bd on bd.id = a.id
LEFT JOIN
cl_customer AS c ON c.id = a.customer_id AND c.deleted = c.deleted
LEFT JOIN
cl_application_packager AS ap ON ap.application_id = a.id AND ap.status != "Declined" AND ap.deleted = "0000-00-00 00:00:00"
WHERE
(a.deleted = "0000-00-00 00:00:00")
GROUP BY
a.id
) AS s
LEFT JOIN
cl_customer_address AS ca2 ON ca2.id = s.address_id AND ca2.deleted = ca2.deleted
WHERE
(ca2.deleted = ca2.deleted OR ca2.deleted IS NULL)
ORDER BY
app_created DESC
LIMIT
0, 100;
Time: 70.382
Rows: 100
The table descriptions are:
CREATE TABLE cl_application (
id int(11) NOT NULL AUTO_INCREMENT,
customer_id int(11) NOT NULL,
product_type tinytext COLLATE utf8_unicode_ci,
introducer_id int(11) NOT NULL,
loan_amount double NOT NULL,
loan_purpose tinytext COLLATE utf8_unicode_ci NOT NULL,
comments text COLLATE utf8_unicode_ci NOT NULL,
security_value double NOT NULL,
property_address_1 tinytext COLLATE utf8_unicode_ci NOT NULL,
property_address_2 tinytext COLLATE utf8_unicode_ci NOT NULL,
property_town_city tinytext COLLATE utf8_unicode_ci NOT NULL,
property_postcode varchar(8) COLLATE utf8_unicode_ci NOT NULL,
property_country tinytext COLLATE utf8_unicode_ci NOT NULL,
application_source tinytext COLLATE utf8_unicode_ci NOT NULL,
`status` enum('WK - Working Lead','APP - Application Taken','ISS - Pack Issued','HOT - Head Of Terms Sent','APU - Application underway','OFI - Offer Issued','DIP - Deal in Progress','DUS - Declined Unsecured/Trying Secured','DUG - Declined Unsecured/Trying Guarantor','COM - Completed Awaiting Payment','PAC - Paid and Completed','TD - Turned Down','DUP - Duplicate application') COLLATE utf8_unicode_ci NOT NULL DEFAULT 'WK - Working Lead',
multi_stage_status text COLLATE utf8_unicode_ci NOT NULL,
owner_id int(11) NOT NULL,
token tinytext COLLATE utf8_unicode_ci NOT NULL,
cache_keyword text COLLATE utf8_unicode_ci NOT NULL,
old_id int(11) NOT NULL,
placed_with tinytext COLLATE utf8_unicode_ci NOT NULL,
submitted datetime NOT NULL,
`database` enum('LV','TD','CD','UL') COLLATE utf8_unicode_ci NOT NULL DEFAULT 'LV',
snoozed datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
created datetime NOT NULL,
edited datetime NOT NULL,
deleted datetime NOT NULL,
PRIMARY KEY (id),
FULLTEXT KEY cache_keyword (cache_keyword)
) ;
CREATE TABLE cl_application_packager (
id int(11) NOT NULL AUTO_INCREMENT,
application_id int(11) NOT NULL,
packager tinytext COLLATE utf8_unicode_ci,
packager_id int(11) DEFAULT NULL,
amount double DEFAULT NULL,
`status` enum('In Progress','Declined','Accepted','Completed') COLLATE utf8_unicode_ci DEFAULT 'In Progress',
commision double DEFAULT NULL,
created datetime NOT NULL,
edited datetime NOT NULL,
deleted datetime NOT NULL,
PRIMARY KEY (id),
KEY application_id (deleted)
) ;
CREATE TABLE cl_application_business_details (
id int(11) NOT NULL AUTO_INCREMENT,
company_number tinytext COLLATE utf8_unicode_ci,
legal_structure enum('Ltd','LLP','Partnership','Sole Trader') COLLATE utf8_unicode_ci DEFAULT NULL,
incorporated date DEFAULT NULL,
created datetime NOT NULL,
edited datetime NOT NULL,
deleted datetime NOT NULL,
PRIMARY KEY (id)
) ;
CREATE TABLE cl_customer (
id int(11) NOT NULL AUTO_INCREMENT,
title enum('Mr','Mrs','Ms','Miss') COLLATE utf8_unicode_ci NOT NULL,
name_first tinytext COLLATE utf8_unicode_ci NOT NULL,
name_last tinytext COLLATE utf8_unicode_ci NOT NULL,
dob date NOT NULL,
marital_status tinytext COLLATE utf8_unicode_ci NOT NULL,
name_company tinytext COLLATE utf8_unicode_ci NOT NULL,
email tinytext COLLATE utf8_unicode_ci NOT NULL,
alt_email tinytext COLLATE utf8_unicode_ci,
contact_number tinytext COLLATE utf8_unicode_ci NOT NULL,
home_phone_number tinytext COLLATE utf8_unicode_ci NOT NULL,
work_phone_number tinytext COLLATE utf8_unicode_ci NOT NULL,
created datetime NOT NULL,
edited datetime NOT NULL,
deleted datetime NOT NULL,
PRIMARY KEY (id)
) ;
CREATE TABLE cl_customer_address (
id int(11) NOT NULL AUTO_INCREMENT,
customer_id int(11) NOT NULL,
application_id int(11) NOT NULL,
house_name tinytext COLLATE utf8_unicode_ci NOT NULL,
house_number tinytext COLLATE utf8_unicode_ci NOT NULL,
address_line_1 tinytext COLLATE utf8_unicode_ci NOT NULL,
address_line_2 tinytext COLLATE utf8_unicode_ci NOT NULL,
address_town tinytext COLLATE utf8_unicode_ci NOT NULL,
address_postcode tinytext COLLATE utf8_unicode_ci NOT NULL,
moved_in date NOT NULL,
vacated date NOT NULL DEFAULT '0000-00-00',
ptcabs tinytext COLLATE utf8_unicode_ci NOT NULL,
resident_status tinytext COLLATE utf8_unicode_ci NOT NULL,
created datetime NOT NULL,
edited datetime NOT NULL,
deleted datetime NOT NULL,
PRIMARY KEY (id),
KEY customer_id (customer_id,deleted)
) ;
mysql> describe cl_application
-> ;
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+-----+---------------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| customer_id | int(11) | NO | | NULL | |
| product_type | tinytext | YES | | NULL | |
| introducer_id | int(11) | NO | | NULL | |
| loan_amount | double | NO | | NULL | |
| loan_purpose | tinytext | NO | | NULL | |
| comments | text | NO | | NULL | |
| security_value | double | NO | | NULL | |
| property_address_1 | tinytext | NO | | NULL | |
| property_address_2 | tinytext | NO | | NULL | |
| property_town_city | tinytext | NO | | NULL | |
| property_postcode | varchar(8) | NO | | NULL | |
| property_country | tinytext | NO | | NULL | |
| application_source | tinytext | NO | | NULL | |
| status | enum('WK - Working Lead','APP - Application Taken','ISS - Pack Issued','HOT - Head Of Terms Sent','APU - Application underway','OFI - Offer Issued','DIP - Deal in Progress','DUS - Declined Unsecured/Trying Secured','DUG - Declined Unsecured/Trying Guarantor','COM - Completed Awaiting Payment','PAC - Paid and Completed','TD - Turned Down','DUP - Duplicate application') | NO | | WK - Working Lead | |
| multi_stage_status | text | NO | | NULL | |
| owner_id | int(11) | NO | | NULL | |
| token | tinytext | NO | | NULL | |
| cache_keyword | text | NO | MUL | NULL | |
| old_id | int(11) | NO | | NULL | |
| placed_with | tinytext | NO | | NULL | |
| submitted | datetime | NO | | NULL | |
| database | enum('LV','TD','CD','UL') | NO | | LV | |
| snoozed | datetime | NO | | 0000-00-00 00:00:00 | |
| created | datetime | NO | | NULL | |
| edited | datetime | NO | | NULL | |
| deleted | datetime | NO | | NULL | |
+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+-----+---------------------+----------------+
27 rows in set (0.00 sec)
--
mysql> describe cl_application_business_details;
+-----------------+-----------------------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+-----------------------------------------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| company_number | tinytext | YES | | NULL | |
| legal_structure | enum('Ltd','LLP','Partnership','Sole Trader') | YES | | NULL | |
| incorporated | date | YES | | NULL | |
| created | datetime | NO | | NULL | |
| edited | datetime | NO | | NULL | |
| deleted | datetime | NO | | NULL | |
+-----------------+-----------------------------------------------+------+-----+---------+----------------+
7 rows in set (0.00 sec)
--
mysql> describe cl_customer;
+-------------------+------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+------------------------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| title | enum('Mr','Mrs','Ms','Miss') | NO | | NULL | |
| name_first | tinytext | NO | | NULL | |
| name_last | tinytext | NO | | NULL | |
| dob | date | NO | | NULL | |
| marital_status | tinytext | NO | | NULL | |
| name_company | tinytext | NO | | NULL | |
| email | tinytext | NO | | NULL | |
| alt_email | tinytext | YES | | NULL | |
| contact_number | tinytext | NO | | NULL | |
| home_phone_number | tinytext | NO | | NULL | |
| work_phone_number | tinytext | NO | | NULL | |
| created | datetime | NO | | NULL | |
| edited | datetime | NO | | NULL | |
| deleted | datetime | NO | | NULL | |
+-------------------+------------------------------+------+-----+---------+----------------+
15 rows in set (0.00 sec)
--
mysql> describe cl_application_packager;
+----------------+-------------------------------------------------------+------+-----+-------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+-------------------------------------------------------+------+-----+-------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| application_id | int(11) | NO | | NULL | |
| packager | tinytext | YES | | NULL | |
| packager_id | int(11) | YES | | NULL | |
| amount | double | YES | | NULL | |
| status | enum('In Progress','Declined','Accepted','Completed') | YES | | In Progress | |
| commision | double | YES | | NULL | |
| created | datetime | NO | | NULL | |
| edited | datetime | NO | | NULL | |
| deleted | datetime | NO | MUL | NULL | |
+----------------+-------------------------------------------------------+------+-----+-------------+----------------+
10 rows in set (0.00 sec)
mysql> describe cl_customer_address;
+------------------+----------+------+-----+------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+----------+------+-----+------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| customer_id | int(11) | NO | MUL | NULL | |
| application_id | int(11) | NO | | NULL | |
| house_name | tinytext | NO | | NULL | |
| house_number | tinytext | NO | | NULL | |
| address_line_1 | tinytext | NO | | NULL | |
| address_line_2 | tinytext | NO | | NULL | |
| address_town | tinytext | NO | | NULL | |
| address_postcode | tinytext | NO | | NULL | |
| moved_in | date | NO | | NULL | |
| vacated | date | NO | | 0000-00-00 | |
| ptcabs | tinytext | NO | | NULL | |
| resident_status | tinytext | NO | | NULL | |
| created | datetime | NO | | NULL | |
| edited | datetime | NO | | NULL | |
| deleted | datetime | NO | | NULL | |
+------------------+----------+------+-----+------------+----------------+
16 rows in set (0.00 sec)
Explain Output:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> ALL 178913936 Using filesort
1 PRIMARY ca2 eq_ref PRIMARY PRIMARY 4 s.address_id 1 Using where
2 DERIVED a ALL PRIMARY, cache_keyword 14494 Using where; Using temporary; Using filesort
2 DERIVED bd eq_ref PRIMARY PRIMARY 4 s-choiceloans.a.id 1
2 DERIVED c eq_ref PRIMARY PRIMARY 4 s-choiceloans.a.customer_id 1 Using where
2 DERIVED ap ALL 12344 Using where; Using join buffer (Block Nested Loop)
3 DEPENDENT SUBQUERY ca1 ref customer_id customer_id 9 func,const 1 Using where; Using filesort
After changing the query based on the answer's follow up:
https://jsbin.com/bonarucisi/2/edit?html,output
The biggest suspect is the inline view (or derived table, in the MySQL venacular.) MySQL is going to materialize the derived table, and then the outer query will run against that.
The other suspect is the correlated subquery in the SELECT list. That's going to be executed for every row returned by the outer query.
Also, the ORDER BY app_created is going to require a Using filesort operation (unless you're running a more recent version of MySQL that might build an index on the derived table.)
There's some odd predicates, e.g. c.deleted = c.deleted
That's going to be true for every row that has a non-NULL value in the deleted column. That's equivalent to c.deleted IS NOT NULL.
And for this: ca2.deleted = ca2.deleted OR ca2.deleted IS NULL, that's always going to be true.
Neither of those should have much impact on performance however.
Run EXPLAIN to see the execution plan.
Absent that, here's a first cut at some recommendations:
Add a covering index on c1_application_packager
CREATE INDEX c1_application_packager_IX2 ON c1_application_packager
(application_id, deleted, status, packager)
;
Add a covering index on cl_customer_address
CREATE INDEX c1_customer_address_IX2 ON c1_customer_address
(customer_id, deleted, created, id, address_postcode, resident_status)
;
And re-write the query to eliminate the derived table. Replace the join to ca2 with the same correlated subquery you used to return the id from c1_customer_address...
SELECT a.id
, c.name_first
, c.name_last
, a.created AS app_created
, a.edited
, a.comments AS custcomment
, c.name_company
, a.loan_amount
, a.product_type
, ap.packager AS placed_with
, a.introducer_id
, a.status
, c.contact_number
, c.email
, a.database
, bd.legal_structure
, ( SELECT ca1.id
FROM cl_customer_address ca1
WHERE ca1.customer_id = a.customer_id
AND ca1.deleted = '0000-00-00 00:00:00'
ORDER BY ca1.customer_id, ca1.deleted, ca1.created DESC
LIMIT 1
) AS address_id
, ( SELECT ca2.address_postcode
FROM cl_customer_address ca2
WHERE ca2.customer_id = a.customer_id
AND ca2.deleted = '0000-00-00 00:00:00'
ORDER BY ca2.customer_id, ca2.deleted, ca2.created DESC
LIMIT 1
) AS address_postcode
, ( SELECT ca3.resident_status
FROM cl_customer_address ca3
WHERE ca3.customer_id = a.customer_id
AND ca3.deleted = '0000-00-00 00:00:00'
ORDER BY ca3.customer_id, ca3.deleted, ca3.created DESC
LIMIT 1
) AS resident_status
FROM cl_application a
LEFT
JOIN cl_application_business_details bd
ON bd.id = a.id
LEFT
JOIN cl_customer c
ON c.id = a.customer_id
AND c.deleted IS NOT NULL
LEFT
JOIN cl_application_packager ap
ON ap.application_id = a.id
AND ap.status != 'Declined'
AND ap.deleted = '0000-00-00 00:00:00'
WHERE a.deleted = '0000-00-00 00:00:00'
GROUP BY a.created, a.id
ORDER BY a.created, a.id
Those correlated subqueries in the SELECT list are going to be executed for every row returned by the query, so that's going to be expensive.
Now we want to see if we can get an index on c1_application table that will help us avoid a Using filesort operation (to satisfy the ORDER BY and the GROUP BY clauses.)
CREATE INDEX c1_application_IX2 ON c1_application
(deleted, created, id)
;
The query relies on a MySQL-specific extension to GROUP BY behavior, not throwing an error due to non-aggregates in the SELECT list which don't appear in the GROUP BY. If there is more than one "matching" row from c1_customer or c1_application_packager, which of the rows is returned out of the GROUP BY operation is indeterminate.
No guarantee that these changes will have a positive impact on performance. (Performance could be a whole lot worse.)
Again, run EXPLAIN to see the execution plan, and adjust from there.
The next big hump is those correlated subqueries. For a shot at decent performance, it's imperative that a suitable index is available. (A proposal for a suitable covering index already specified above.)
The next cut would be eliminate the correlated subqueries from the SELECT list, If the id column from c1_customer_address was being returned as a means to get the address_postcode and resident_status, that first correlated subquery could be eliminated.
FOLLOWUP
Removing the correlated subqueries... adding an inline view lc to get the latest created date from c1_customer_address (for each customer_id), and another join to c1_customer_address to retrieve the rows with that latest created date (for each customer).
The lc inline view introduces a "derived table", which can be expensive for large sets, but this may be faster than using correlated subqueries.
SELECT a.id
, c.name_first
, c.name_last
, a.created AS app_created
, a.edited
, a.comments AS custcomment
, c.name_company
, a.loan_amount
, a.product_type
, ap.packager AS placed_with
, a.introducer_id
, a.status
, c.contact_number
, c.email
, a.database
, bd.legal_structure
, ca.id AS address_id
, ca.address_postcode
, ca.resident_status
FROM cl_application a
LEFT
JOIN cl_application_business_details bd
ON bd.id = a.id
LEFT
JOIN cl_customer c
ON c.id = a.customer_id
AND c.deleted IS NOT NULL
LEFT
JOIN cl_application_packager ap
ON ap.application_id = a.id
AND ap.status != 'Declined'
AND ap.deleted = '0000-00-00 00:00:00'
LEFT
JOIN ( SELECT ca1.customer_id
, MAX(ca1.created) AS latest_created
FROM cl_customer_address ca1
WHERE ca1.deleted = '0000-00-00 00:00:00'
GROUP BY ca1.customer_id
) lc
ON lc.customer_id = a.customer_id
LEFT
JOIN cl_customer_address ca
ON ca.customer_id = lc.customer_id
AND ca.created = lc.latest_created
AND ca.deleted = '0000-00-00 00:00:00'
WHERE a.deleted = '0000-00-00 00:00:00'
GROUP BY a.created, a.id
ORDER BY a.created, a.id

Select all categories with latest post, user, and topic information

I am working on a custom forum for a web project. I have categories with id, topics with id and category, and posts with id and topic and user id. What I'm trying to do is display a list of the categories with data from the categories table along with data from the posts table for the newest post in that category, data for that posts's associated user, as well as some data for the topic associated with that latest post.
I've been banging my head on the wall trying to figure the query out, but I just don't have a good enough understanding of complex mysql queries to know what pattern or technique to use here. Here is what I have so far:
SELECT u1.*, fp1.*, ft1.*, fc1.* from forum_posts AS fp1
LEFT JOIN users AS u1 ON u1.id = fp1.post_by
LEFT JOIN forum_topics AS ft1 on ft1.id = fp1.post_topic
LEFT JOIN forum_categories AS fc1 on fc1.id = ft1.topic_cat
GROUP BY fc1.id
ORDER BY fp1.id ASC;
But this does not return the results I'm looking for. The problem is in trying to get the newest post for each category and the associate topic for that post.
Here is the DB structure for each table:
forum_categories
+-----------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| cat_name | varchar(255) | NO | UNI | NULL | |
| cat_description | varchar(500) | NO | | NULL | |
| cat_views | bigint(20) unsigned | YES | | 0 | |
| status | tinyint(1) | NO | | 1 | |
+-----------------+---------------------+------+-----+---------+----------------+
forum_topics
+---------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| topic_subject | varchar(255) | NO | | NULL | |
| topic_date | datetime | NO | | NULL | |
| topic_cat | bigint(20) unsigned | NO | MUL | NULL | |
| topic_by | bigint(20) unsigned | NO | MUL | NULL | |
| topic_views | bigint(20) unsigned | YES | | 0 | |
+---------------+---------------------+------+-----+---------+----------------+
forum_posts
+--------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| post_content | text | NO | | NULL | |
| post_date | datetime | NO | | NULL | |
| post_topic | bigint(20) unsigned | NO | MUL | NULL | |
| post_by | bigint(20) unsigned | NO | MUL | NULL | |
+--------------+---------------------+------+-----+---------+----------------+
users
+-----------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| user_type | varchar(50) | YES | | NULL | |
| email | varchar(255) | NO | UNI | NULL | |
| username | varchar(255) | YES | UNI | NULL | |
| password | char(60) | NO | | NULL | |
| image | text | YES | | NULL | |
| status | tinyint(1) | NO | | 1 | |
+-----------+---------------------+------+-----+---------+----------------+
Here is an image of the output I'm trying to achieve. "Categories" shows the data from the forum_categories table and under "Recent" is the user, post and topic data for the latest post:
Please help me out. Thank you.
While it's simple enough to join all the tables together to work out the topic, category and user for each post, you also need one additional step - you need to join to a subquery that retrieves the max(post_id) per category.
Here's one way you can do that:
select fc.cat_name, fc.cat_description, fc.cat_views, u.username, fp.post_date, ft.topic_subject
from forum_categories fc
inner join forum_topics ft
on fc.id = ft.topic_cat
inner join forum_posts fp
on fp.post_topic = ft.id
inner join users u
on fp.post_by = u.id
inner join (
select topic_cat, max(fp.id) most_recent_post
from forum_topics ft
inner join forum_posts fp
on fp.post_topic = ft.id
group by topic_cat
) q
on q.topic_cat = ft.topic_cat
and fp.id = q.most_recent_post;
There's a demo you can play with here: http://sqlfiddle.com/#!9/3736b/1

MySQL query too slow, 70k rows 12 sec

This query runs for 12+ seconds on VPS. It joins 3 tables. Only first one "topcics" has about 70k rows, others are about 20 and "post_cc" about 1500.
SELECT topics.*, employee.username, accounts.ac_name, accounts.ac_mail
FROM topics
INNER JOIN employee ON employee.id_user = topics.id_owner
INNER JOIN accounts ON accounts.id_account = topics.id_account
WHERE topics.status IN ('1','3') AND ( topics.id_owner IN (12, 5) OR topics.id_post IN
(SELECT DISTINCT(id_post) FROM post_cc WHERE id_employee IN (12, 5) ) )
ORDER BY topics.creationdate DESC LIMIT 0,25
I have already tried (without any improvement) to remove subquery and first "employee" join. If I remove "accounts" join, query runs under 0.1 sec, but all tables data are needed for sorting purpose during paging.
Explain:
+----+--------------------+------------+-----------------+-----------------------+---------+---------+-----------------+-------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+------------+-----------------+-----------------------+---------+---------+-----------------+-------+----------------------------------------------+
| 1 | PRIMARY | topics | ALL | id_owner,id_account | NULL | NULL | NULL | 75069 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | accounts | ALL | PRIMARY | NULL | NULL | NULL | 5 | Using where; Using join buffer |
| 1 | PRIMARY | employee | eq_ref | PRIMARY | PRIMARY | 3 | topics.st_owner | 1 | Using where |
| 2 | DEPENDENT SUBQUERY | post_cc | unique_subquery | PRIMARY | PRIMARY | 8 | func,const | 1 | Using index; Using where |
+----+--------------------+------------+-----------------+-----------------------+---------+---------+-----------------+-------+----------------------------------------------+
I have added suggested keys as index, it improved time for 2 sec., but it's still too slow.
Shortened tables:
topics
+--------------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+---------------------+------+-----+---------+----------------+
| id_post | int(10) unsigned | NO | PRI | NULL | auto_increment |
| id_account | int(10) unsigned | YES | MUL | 0 | |
| mail | varchar(256) | YES | MUL | NULL | |
| from_name | varchar(512) | YES | | NULL | |
| title | varchar(512) | YES | | NULL | |
| content | text | YES | | NULL | |
| id_owner | int(10) unsigned | YES | MUL | NULL | |
| creationdate | datetime | YES | | NULL | |
+--------------------+---------------------+------+-----+---------+----------------+
employee
+---------------------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+-----------------------+------+-----+---------+----------------+
| id_employee | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| id_user | mediumint(8) unsigned | NO | | NULL | |
| id_owner | tinyint(1) | YES | | 0 | |
| active | tinyint(1) | YES | | 1 | |
| username | varchar(64) | YES | | NULL | |
| email | varchar(128) | YES | | NULL | |
+---------------------+-----------------------+------+-----+---------+----------------+
accounts
+----------------------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------------+---------------------+------+-----+---------+----------------+
| id_account | int(10) unsigned | NO | PRI | NULL | auto_increment |
| ac_mail | int(10) unsigned | YES | UNI | NULL | |
| ac_name | varchar(512) | YES | | NULL | |
| last_sync_time | datetime | YES | | NULL | |
+----------------------------+---------------------+------+-----+---------+----------------+
post_cc
+------------------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+---------------------+------+-----+---------+-------+
| id_post | int(10) unsigned | NO | PRI | NULL | |
| id_employee | int(10) unsigned | NO | PRI | NULL | |
| notifications | tinyint(3) unsigned | YES | | 1 | |
+------------------------+---------------------+------+-----+---------+-------+
One likely suspect is that DEPENDENT SUBQUERY.
MySQL is processing that subquery for each row returned by the outer query (which isn't already filtered out by some other predicate.
To improve performance, consider re-writing that either as a JOIN operation or an EXISTS predicate.
To replace that with a JOIN operation, that will need to be an OUTER JOIN (rather than an INNER JOIN) because of the OR in the predicate.
As an example of one way to do that:
SELECT topics.*
, employee.username
, accounts.ac_name
, accounts.ac_mail
FROM topics
JOIN employee ON employee.id_user = topics.id_owner
JOIN accounts ON accounts.id_account = topics.id_account
LEFT
JOIN ( SELECT DISTINCT q.id_post
FROM post_cc q
WHERE q.id_employee IN (12, 5)
) p
ON p.id_post = topics.id_post
WHERE topics.status IN ('1','3')
AND ( topics.id_owner IN (12, 5)
OR p.id_post IS NOT NULL
)
ORDER BY topics.creationdate DESC LIMIT 0,25
I recommend you run an EXPLAIN on that, and see how that performs.
Another option is to consider an EXISTS predicate. Occassinally we can get this to perform better, but often times not.
SELECT topics.*
, employee.username
, accounts.ac_name
, accounts.ac_mail
FROM topics
JOIN employee ON employee.id_user = topics.id_owner
JOIN accounts ON accounts.id_account = topics.id_account
WHERE topics.status IN ('1','3')
AND ( topics.id_owner IN (12, 5)
OR EXISTS ( SELECT 1
FROM post_cc q
WHERE q.id_employee IN (12, 5)
AND q.id_post = topics.id_post
)
)
ORDER BY topics.creationdate DESC LIMIT 0,25
For performance, that's going to almost require a suitable covering index for the subquery in the EXISTS clause, for example:
ON post_cc (id_post, id_employee)
You can try running an EXPLAIN and see how that performs as well.
We see that MySQL isn't using an index on the topics table.
We might get MySQL to avoid an expensive "Using filesort" operation if we had an index with a leading column of creationdate.
And part of the problem is likely that OR in the predicate. We might try re-writing that query as two separate queries, and combining them with a UNION ALL set operation. But if we do that, we'd really like to see an index on topic being used (we probably won't improve performance by incurring two scans of 70,000 rows.
SELECT topics.*
, employee.username
, accounts.ac_name
, accounts.ac_mail
FROM topics
JOIN employee ON employee.id_user = topics.id_owner
JOIN accounts ON accounts.id_account = topics.id_account
WHERE topics.status IN ('1','3')
AND topics.id_owner IN (12, 5)
UNION ALL
SELECT topics.*
, employee.username
, accounts.ac_name
, accounts.ac_mail
FROM topics
JOIN employee ON employee.id_user = topics.id_owner
JOIN accounts ON accounts.id_account = topics.id_account
JOIN ( SELECT DISTINCT q.id_post
FROM post_cc q
WHERE q.id_employee IN (12, 5)
) p
ON p.id_post = topics.id_post
WHERE topics.status IN ('1','3')
AND ( topics.id_owner NOT IN (12, 5) OR topics.id_owner IS NULL )
ORDER BY 8 DESC LIMIT 0,25
With a query of that form, we're more likely to get MySQL use a suitable index on the topics table,
... ON topics (id_owner, status)
... ON topics (id_post, status, id_owner)
Why don't you use left join for post_cc table, and then use condition?
Something like this.
SELECT topics.*,
employee.username,
accounts.ac_name,
accounts.ac_mail
FROM topics
INNER JOIN employee ON employee.id_user = topics.id_owner
INNER JOIN accounts ON accounts.id_account = topics.id_account
LEFT JOIN post_cc ON id_employee IN (12, 5)
WHERE topics.status IN ('1','3')
AND ( topics.id_owner IN (12, 5) OR topics.id_post IN (post_cc.post_id)
ORDER BY topics.creationdate DESC LIMIT 0,25;
The culprit:
75069 (rows) | Using where; Using temporary; Using filesort
That's what we need to get rid of.
Possible solution: add an index on topics.creationdate.
As a side note, the query also has conditions on id_post, st_owner, and status, therefore a composite index on topics(creationdate, id_post, st_owner, status) (or any permutation of the last three columns -- test with your data set) might help even further. However, your query seems to pull most of your table anyways, so I expect a simple index will suffice.

Mysql left outer join column does not exist - CodeIgniter DataMapper ORM

I'm running the following query, which is being generated by CodeIgniter's DataMapper ORM as I try to query a deep relationship of the form order_product/order/order_status:
SELECT `order_products`.*
FROM (`order_products`)
LEFT OUTER JOIN `orders` orders ON `orders`.`id` = `order_products`.`order_id`
LEFT OUTER JOIN `order_statuses` order_order_statuses ON `order_order_statuses`.`id` = `orders`.`order_status_id`
WHERE `order_products`.`sku` IN ('SYB-SAMPLETEST8', 'Copy of SYB-SAMPLE#B')
AND `order_products`.`deleted` = 0
AND `orders`.`ext_created_on` >= '2014-05-05'
AND `order_products`.`deleted` = 0
AND `orders`.`ext_created_on` <= '2014-05-21'
AND `orders`.`deleted` = 0
AND `order_statuses`.`deleted` = 0
AND `order_order_statuses`.`sales_data` = 1
I'm receiving the following error:
ERROR 1054 (42S22): Unknown column 'order_statuses.deleted' in 'where clause'
My schema for order_statuses table is as follows:
+-------------+--------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+-------------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| ext_id | int(11) | YES | | NULL | |
| ext_type_id | int(11) | YES | | NULL | |
| name | varchar(255) | YES | | NULL | |
| sales_data | tinyint(1) | YES | | 0 | |
| deleted | tinyint(1) | NO | | 0 | |
| created_on | timestamp | NO | | CURRENT_TIMESTAMP | |
+-------------+--------------+------+-----+-------------------+----------------+
Why am I receiving this error even though the deleted column exists?
Because You are aliasing it to order_order_statuses.

Categories