so I have this query
SELECT a.`title`,a.`id`,a.`numvol`,a.`numepi`,a.`release_date`,
(SELECT COUNT(id) FROM `Release` r WHERE r.`article_id`=a.`id`) AS `num_rows`,
(SELECT COUNT(id) FROM `Article_views` av WHERE av.`article_id`=a.`id`) AS `num_rows2`
FROM `Article` a WHERE a.`type` = 'ani' ORDER BY a.`title` ASC
The first load takes up to 5 secs and if I do a refresh it will take about 0.001 sec, is there a way to uniform the loading time?
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY a ALL NULL NULL NULL NULL 567 Using where; Using filesort
3 DEPENDENT SUBQUERY av ALL NULL NULL NULL NULL 5301 Using where
2 DEPENDENT SUBQUERY r ALL NULL NULL NULL NULL 11717 Using where
I tried to do it with join but it didn't work at all so I gave up this way...
Solution
use barmar query. way better :) (and indexes -,-')
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 536 Using temporary; Using filesort
1 PRIMARY a eq_ref PRIMARY PRIMARY 4 r.art.. 1 Using where
1 PRIMARY <derived3> ALL NULL NULL NULL NULL 574 Using where; Using join buffer
3 DERIVED Article_views index NULL article_id 4 NULL 5301 Using index
2 DERIVED Release index NULL article_id 4 NULL 11717 Using index
Thanks guys for your time and the solution :) I guess I need to redo a good part of this old project ahah :)
Try this query instead:
SELECT a.`title`,a.`id`,a.`numvol`,a.`numepi`,a.`release_date`, `num_rows`, `num_rows2`
FROM `Article` a
JOIN (SELECT article_id, COUNT(*) AS num_rows
FROM Release
GROUP BY article_id) r
ON r.article_id = a.id
JOIN (SELECT article_id, COUNT(*) AS num_rows2
FROM Article_views
GROUP BY article_id) av
ON av.article_id = a.id
WHERE a.`type` = 'ani'
ORDER BY a.`title` ASC
In my experience, JOINs are faster than correlated subqueries.
For performance, make sure you have indexes on Release.article_id and Article_views.article_id.
I guess, 2nd try is benefit from SQL QUERY CACHE. I wonder if adding SQL_NO_CACHE, every try took 5 secs?
SELECT SQL_NO_CACHE a.`title`,a.`id`,a.`numvol`,a.`numepi`,a.`release_date`,
....
INDEXES
Oops. you have no relevant INDEX. could you add following indexes?
ALTER TABLE Article ADD INDEX(type);
ALTER TABLE Release ADD INDEX(article_id);
ALTER TABLE Article_views ADD INDEX(article_id);
More Efficient Query
And your Query converted into JOIN. I guess this is much faster than yours. Assuming every Article has Release and Article_views
SELECT a.`title`,a.`id`,a.`numvol`,a.`numepi`,a.`release_date`,
COUNT(r.id) AS `num_rows`,
COUNT(av.id) AS `num_rows2`
FROM `Article` a JOIN Release r ON r.`article_id`=a.`id`
JOIN Article_views av ON av.`article_id`=a.`id`
WHERE a.`type` = 'ani'
GROUP BY a.title, a.id, a.numvol, a.numepi, a.release_date
ORDER BY a.`title` ASC;
That significant improvement on query latency is due to internal MySQL cache functioning.
After the first execution of the query the result set is cached in RAM, so the results second query which just matches the previous, are immediately taken from the RAM without HDD accesses.
There are different points of view about MySQL internal cache, and experts often recommend to disable it in highload production environments using memecached, Redis or some other caching layer instead.
But definitely you should try to optimize performance of your query with caching turned off - 5 seconds is extremely slow.
Try not to use subqueries, cause MySQL optimizer is not performant on them.
Store values of the counters (results of count()) in separate table and update them properly. Then you can use just the counters values in your query without performing heavy database request each time.
Create indexes, for example, for type field.
Use EXPLAIN for further optimizations
Related
I've got a serious problem. Our Intranet is getting slower and slower. One of the mainreasons seems to be a slow mysql-query (it appears in the slow-query.log).
That query is asked every time an intranet-site is opened.
It looks like this:
SELECT w.Datetime, w.User_ID, w.Status, e.Lastname
FROM worktimes AS w
INNER JOIN employees AS e ON w.User_ID=e.ID
RIGHT JOIN (SELECT max(Datetime) AS Datetime, User_ID
FROM worktimes
WHERE Datetime>".$today." // variable of today 0.00 o'clock
AND Location='".$llocation['ID']."' // variable of one of 9 locations
GROUP BY User_ID) AS v
ON v.User_ID=w.User_ID AND w.Datetime=v.Datetime
ORDER BY e.Lastname;
The worktimes-table is somewhat greater with up to 200k rows (momentary 90k to testing reasons) and 13 columns. The whole query goes through a loop with 3 to 9 cycles.Has someone an idea how to make the queries faster?
edit: As wished here is the EXPLAIN-result.
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 44006 Using temporary; Using filesort
1 PRIMARY w ALL NULL NULL NULL NULL 92378 Using where
1 PRIMARY e eq_ref PRIMARY,ID PRIMARY 4 ais_v1.w.User_ID 1 NULL
2 DERIVED worktimes ref Location Location 767 const 44006 Using index condition; Using where; Using temporary; Using filesort
The w table needs an index on INDEX(Location, Datetime). This should improve the performance.
you dont need to use worktimes twice
you can do it as follows:
SELECT max(w.Datetime) AS Datetime, w.User_ID, w.User_ID, w.Status, e.Lastname
FROM worktimes w left outer join employees e
on e.User_ID=w.User_ID
and w.Datetime>".$today." // variable of today 0.00 o'clock
AND w.Location='".$llocation['ID']."' // variable of one of 9 locations
GROUP BY w.User_ID
ORDER BY e.Lastname;
it will run faster than your existing query
I have a fairly simple query which runs okay when I test it in phpMyAdmin:
SELECT
c.customers_id,
c.customers_cid,
c.customers_gender,
c.customers_firstname,
c.customers_lastname,
c.customers_email_address,
c.customers_telephone,
c.customers_date_added,
ab.entry_company,
ab.entry_street_address,
ab.entry_postcode,
ab.entry_city,
COUNT(o.customers_id) AS orders_number,
SUM(ot.value) AS totalvalue,
mb.bonus_points
FROM
orders AS o,
orders_total AS ot,
customers AS c,
address_book AS ab,
module_bonus AS mb
WHERE
c.customers_id = o.customers_id
AND c.customers_default_address_id = ab.address_book_id
AND c.customers_id = mb.customers_id
AND o.orders_id = ot.orders_id
AND ot.class = 'ot_subtotal'
** AND c.customers_gender = 'm' AND c.customers_lastname LIKE 'Famlex'
GROUP BY o.customers_id
The row marked with ** changes depending on filtering settings of the application making the query.
Now, when I test this in phpMyAdmin, the query takes a couple of seconds to run (which is fine, since there are thousands of entries and, as far as I know, when using COUNTs and SUMs indexes don't help) and the results are perfect, but when I run the exact same query in PHP (echoed before running), the MySQL thread loads a core to 100% and doesn't stop until I kill it.
If I strip the extra stuff to calculate the COUNT and SUM, the query finishes but the results are useless to me.
EXPLAIN:
1 SIMPLE mb ALL NULL NULL NULL NULL 48713 Using temporary; Using filesort
1 SIMPLE ot ALL idx_orders_total_orders_id NULL NULL NULL 811725 Using where
1 SIMPLE o eq_ref PRIMARY PRIMARY 4 db.ot.orders_id 1 Using where
1 SIMPLE c eq_ref PRIMARY PRIMARY 4 db.o.customers_id 1 Using where
1 SIMPLE ab eq_ref PRIMARY PRIMARY 4 db.c.customers_default_address_id 1
EXPLAIN after applying indexes and using joins:
1 SIMPLE c ref PRIMARY,search_str_idx search_str_idx 98 const 1 Using where; Using temporary; Using filesort
1 SIMPLE mb ALL NULL NULL NULL NULL 48713 Using where
1 SIMPLE ab eq_ref PRIMARY PRIMARY 4 db.c.customers_default_address_id 1
1 SIMPLE ot ref idx_orders_total_orders_id,class class 98 const 157004 Using where
1 SIMPLE o eq_ref PRIMARY PRIMARY 4 db.ot.orders_id 1 Using where
Use explicit join instead of implicit
SELECT
c.customers_id,
c.customers_cid,
c.customers_gender,
c.customers_firstname,
c.customers_lastname,
c.customers_email_address,
c.customers_telephone,
c.customers_date_added,
ab.entry_company,
ab.entry_street_address,
ab.entry_postcode,
ab.entry_city,
COUNT(o.customers_id) AS orders_number,
SUM(ot.value) AS totalvalue,
mb.bonus_points
FROM
orders o
join orders_total ot on o.orders_id = ot.orders_id
join customers c on c.customers_id = o.customers_id
join address_book ab on c.customers_default_address_id = ab.address_book_id
join module_bonus mb on c.customers_id = mb.customers_id
where
ot.class = 'ot_subtotal'
c.customers_gender = 'm'
AND c.customers_lastname = 'Famlex'
GROUP BY o.customers_id
Assuming all the joining keys are also primary key of those tables viz:
o.orders_id, c.customers_id, ab.address_book_id
You will need to add the following indexes if they are not added already
alter table orders add index customers_id_idx(customers_id);
alter table module_bonus add index customers_id_idx(customers_id);
alter table orders_total add index orders_id_idx(orders_id);
alter table orders_total add index orders_class_idx(class);
alter table customers add index search_str_idx(customers_gender,customers_lastname);
Make sure to take a backup of the tables before applying indexes.
Can you share SQL dump of your record so that I can take a look ?
phpMyAdmin automatically adds limit clause to the select queries, that is why I think you are getting the impression that in phpMyAdmin query is running fine but not via you PHP script.
Try to add explicit limit clause to the query say limit 0, 1000 in phpMyAdmin before running and see if that makes performance of phpMyAdmin slower.
I have a query with 2 sub selects that in the PHP page takes upwards of 20 seconds to process (just to populate a dropdown box!) but when I run the query via phpmyadmin it returns all its results in 0.4 seconds
SELECT ID, SupplierName
FROM tblsuppliers
WHERE ID
IN (
SELECT DISTINCT Supplier
FROM tblmovements
WHERE SuppIsDisputed =1
)
OR ID
IN (
SELECT DISTINCT UsedSupplier
FROM tblsundries
WHERE SuppIsDisputed =1
)
ORDER BY SupplierName ASC
So I thought well sub selects and IN()'s are slow and expensive I'll rewrite to joins.
SELECT
DISTINCT
tblsuppliers.ID,
tblsuppliers.SupplierName
FROM
tblsuppliers
LEFT JOIN tblmovements ON tblsuppliers.ID=tblmovements.Supplier
LEFT JOIN tblsundries ON tblsuppliers.ID=tblsundries.UsedSupplier
WHERE
tblmovements.SuppIsDisputed=1
OR
tblsundries.SuppIsDisputed=1
ORDER BY tblsuppliers.SupplierName ASC
Using GROUP BY tblsuppliers.ID the query takes upwards of 60 seconds even ran in phpmyadmin (page times out) but using distinct it completes but in 20 seconds so still slower than the sub selects when done directly against the DB
So I thought I'd profile the queries.
The sub select query gives me the following when profiled.
Status Time
starting 0.000042
checking permissions0.000004
checking permissions0.000001
checking permissions0.000002
Opening tables 0.000025
System lock 0.000006
init 0.000032
optimizing 0.000006
statistics 0.000007
preparing 0.000007
executing 0.000001
Sorting result 0.000052
optimizing 0.000006
statistics 0.000008
preparing 0.003540
optimizing 0.000012
statistics 0.000010
preparing 0.408007
Sending data 0.000031
end 0.000004
query end 0.000006
closing tables 0.000011
freeing items 0.000085
logging slow query 0.000002
cleaning up 0.000001
Explains why its fast in phpmyadmin but doesn't explain why it takes 20 seconds in PHP to run the same query!
The join query gives me the following when profiled.
Status Time
starting 0.000045
checking permissions0.000003
checking permissions0.000001
checking permissions0.000005
Opening tables 0.000027
System lock 0.000006
init 0.000027
optimizing 0.000009
statistics 0.000021
preparing 0.000011
Creating tmp table 0.000132
executing 0.000002
Copying to tmp table20.071386
Sorting result 0.000090
Sending data 0.000019
end 0.000002
removing tmp table 0.000007
end 0.000003
query end 0.000003
closing tables 0.000010
freeing items 0.000087
logging slow query 0.000002
logging slow query 0.000001
cleaning up 0.000001
I found that a little wierd it explains the 20 second process time when using distinct. Not sure how I can improve the time copying to the tmp table?
Based off the above I have the following questions.
Is there a better way of rewriting the subselect query to be join
based or generally faster?
Why is the sub select taking only 0.4 seconds when ran in phpmyadmin
/ in mysqlclient but taking ages in php?
Is there any way to stop (even if its server setting alterations)
the copying to tmp table for the join based query taking 20 seconds
+.
Am I just being an idiot?
For reference the block of PHP that generates the drop down is below (using the sub select query)
$supplier = $db->query("SELECT ID,SupplierName FROM tblsuppliers WHERE ID IN(SELECT DISTINCT Supplier FROM tblmovements WHERE SuppIsDisputed=1) OR ID IN(SELECT DISTINCT UsedSupplier FROM tblsundries WHERE SuppIsDisputed=1) ORDER BY SupplierName ASC",ENABLE_DEBUG);
if ($db->numRows($supplier)>0) {
while ($suppliers = $db->fetchNextObject($supplier)) {
$selected = ($suppliers->ID==$_POST['filter']) ? "selected=\"selected\"" : "" ;
echo "<option value=\"".stripslashes($suppliers->ID)."\" $selected>".stripslashes($suppliers->SupplierName)."</option>";
}
}
EDIT: explains on queries.
Columns with INDEX and column data types
tblmovements.Supplier - type int(10)
tblsundries.UsedSupplier - type int(10)
tblsuppliers.SupplierName - type varchar(200)
tblsuppliers.ID - type int(10) auto increment primary key
The following columns have no index on but are tinyint(1) values of 0 or 1.
tblmovements.SuppIsDisputed=1
tblsundries.SuppIsDisputed=1
Sub select Explain:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY tblsuppliers ALL NULL NULL NULL NULL 1341 Using where; Using filesort
3 DEPENDENT SUBQUERY tblsundries index_subquery UsedSupplier UsedSupplier 4 func 22 Using where
2 DEPENDENT SUBQUERY tblmovements index_subquery Supplier Supplier 8 func 157 Using where
Join Explain:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE tblsuppliers ALL NULL NULL NULL NULL 1403 Using temporary; Using filesort
1 SIMPLE tblmovements ref Supplier Supplier 8 ggdashboard.tblsuppliers.ID 157 Distinct
1 SIMPLE tblsundries ref UsedSupplier UsedSupplier 4 ggdashboard.tblsuppliers.ID 22 Using where; Distinct
Looking at the row counts from explain output it seems there are very few rows having SuppIsDisputed = 1 in both tables so idexes on these columns would be very selective for this condition. I'd try adding these indexes and rewriting your query as union of two idependent joins, which would result in a filesort only for the ready rowset after applying union, to sort rows by SupplierName, and it would easily fit in the memory because of low row count.
So run this first:
ALTER TABLE tblmovements ADD INDEX m_disputed( SuppIsDisputed );
ALTER TABLE tblsundries ADD INDEX s_disputed( SuppIsDisputed );
and then use this query in your code:
( SELECT DISTINCT s.Id, s.SupplierName FROM tblsuppliers s
JOIN tblmovements m ON s.ID = m.Supplier
WHERE m.SuppIsDisputed = 1 )
UNION
( SELECT DISTINCT s.ID, s.SupplierName FROM tblsuppliers s
JOIN tblsundries sd ON s.ID = sd.UsedSupplier
WHERE sd.SuppIsDisputed = 1 )
ORDER BY SupplierName
This is your query:
SELECT ID, SupplierName
FROM tblsuppliers
WHERE ID IN (SELECT DISTINCT Supplier
FROM tblmovements
WHERE SuppIsDisputed = 1
) OR
ID IN (SELECT DISTINCT UsedSupplier
FROM tblsundries
WHERE SuppIsDisputed = 1
)
ORDER BY SupplierName ASC;
I'm going to propose a three-part solution for increasing performance. First, rewrite the query to use exists rather than in:
SELECT ID, SupplierName
FROM tblsuppliers s
WHERE EXISTS (SELECT 1
FROM tblmovements m
WHERE m.SuppIsDisputed = 1 and s.id = m.Supplier
) OR
EXISTS (SELECT 1
FROM tblsundries su
WHERE su.SuppIsDisputed = 1 and s.id = su.UsedSupplier
)
ORDER BY SupplierName ASC;
Second, add indexes on the tables used in the subqueries to make the lookups faster: tblmovements(Supplier, SuppIsDisputed) and tblsundries(UsedSupplier, SuppIsDisputed).
Finally, add an index on the outer query to avoid the final sort: tblsuppliers(SupplierName, id).
i faced the problem on query processing for large amount of data in MySql. i need to fetch data into more than four tables by join function . the query runs very slowly on server .
how to optimize the processing time for multiple join queries.
I am currently using the innodb engine . It is ok for bulk data table .
i have tables like
Data in those tables exam,students,subjects,subject_tests,subject_test_mark,total_subject,
i get all students record of current exam .
This will previously process by multiple for loops . continuous db access . That a reason to slow down my process . how to avoid these scenarios by SQL . Suggest me some ideas that are welcome .
i have some doubts on this . Table engine innodb is fine or not . Indexing is support for this process or not ?
SELECT `sub`.student_id,
((sum( `sub`.total / `main`.max_total )) / count( sub.id )) *10 AS percent,
GROUP_CONCAT((sub.total / main.max_total) *10 ORDER BY (sub.total / main.max_total) *10 DESC SEPARATOR ',' ) AS marks
FROM
`cmark_batch` AS `main`
LEFT JOIN `cmark_para_total` AS `sub`
ON `sub`.mark_batch_id = `main`.id
WHERE main.batch_id =29
AND main.term =4
AND main.subject_id =64
AND main.status = 'A'
GROUP BY student_id
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE main ref search search 8 const 85 Using index condition; Using where; Using temporary; Using filesort
1 SIMPLE sub ref finder finder 8 newlive.main.id 14 NULL
SELECT t1.mark_batch_id, t1.id, t1.param_name, t1.max_mark,t2.student, t2.mark
FROM `cmark_parameter` AS t1
LEFT JOIN `cmark_parameter_mark` AS t2 ON t1.id = t2.mark_parameter_id
WHERE t1.mark_batch_id
IN (621,620,623,622)
AND t1.status = 'A'
AND t2.status = 'A'
ORDER BY `t2`.`student` ASC
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE t2 ALL NULL NULL NULL NULL 78835 Using where; Using filesort
1 SIMPLE t1 eq_ref PRIMARY PRIMARY 8 newlive.t2.cmark_parameter_id 1 Using where
SELECT t1.student_id, t1.mark_batch_id, t1.total
FROM `cmark_para_total` AS t1
WHERE t1.mark_batch_id
IN (621,620,623,622)
AND t1.status = 'A'
ORDER BY `t1`.`id` ASC
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE t1 range finder finder 8 NULL 111 Using index condition; Using where; Using filesort
You can use database caching to reduce the time of loading complex queries.
In the Mysql manual is a description how you can enable database caching:
http://dev.mysql.com/doc/refman/5.1/en/query-cache.html
In your query you have to add the attribute SQL_CACHE to enable caching, example:
SELECT SQL_CACHE id, name FROM customer;
Use the keyword 'EXPLAIN' to determine what is causing the slowness and what you can do to optimize.
More here:
https://dev.mysql.com/doc/refman/5.0/en/using-explain.html
I'm having real problems with my Mysql statement, I need to join a few tables together, query them and order by the average of values from another table. This is what I have...
SELECT
ROUND(avg(re.rating), 1)AS avg_rating,
s.staff_id, s.full_name, s.mobile, s.telephone, s.email, s.drive
FROM staff s
INNER JOIN staff_homes sh
ON s.staff_id=sh.staff_id
INNER JOIN staff_positions sp
ON s.staff_id=sp.staff_id
INNER JOIN reliability re
ON s.staff_id=re.staff_id
INNER JOIN availability ua
ON s.staff_id=ua.staff_id
GROUP BY staff_id
ORDER BY avg_rating DESC
Now I believe this to work although I am getting this error "The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay".
I think this means that I have too many joins and because it is shared hosting it won't allow large queries to run I don't know.
What I would like to know is exactly what the error means (I have googled it but I don't understand the answers) and how I can work round it by maybe making my query more efficient?
Any help would be appreciated. Thanks
EDIT:
The reason I need the joins is so I can query the tables based on a search function like so...
SELECT
ROUND(avg(re.rating), 1)AS avg_rating
, s.staff_id, s.full_name, s.mobile, s.telephone, s.email, s.drive
FROM staff s
INNER JOIN staff_homes sh
ON s.staff_id=sh.staff_id
INNER JOIN staff_positions sp
ON s.staff_id=sp.staff_id
INNER JOIN reliability re
ON s.staff_id=re.staff_id
INNER JOIN availability ua
ON s.staff_id=ua.staff_id
WHERE s.full_name LIKE '%'
AND s.drive = '1'
AND sh.home_id = '3'
AND sh.can_work = '1'
AND sp.position_id = '3'
AND sp.can_work = '1'
GROUP BY staff_id
ORDER BY avg_rating DESC
EDIT 2
This was the result of my explain. Also I'm not great with MYSQL how would I set up foreign keys?
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE ua ALL NULL NULL NULL NULL 14 Using temporary; Using filesort
1 SIMPLE re ALL NULL NULL NULL NULL 50 Using where; Using join buffer
1 SIMPLE sp ALL NULL NULL NULL NULL 84 Using where; Using join buffer
1 SIMPLE sh ALL NULL NULL NULL NULL 126 Using where; Using join buffer
1 SIMPLE s eq_ref PRIMARY PRIMARY 4 web106-prestwick.ua.staff_id 1
EDIT 3: Thanks lc, it was my foreign keys, they were not set up correctly. Problem sorted
Maybe you should use more and/or better indexes on the tables.
According to the db you're using, the optimization may be faster or not with subqueries.
There may be 2 bottlenecks on your query:
try to remove the average function of your query. If your query speeds up, try to replace it with a subquery and see what happens.
The multiple joins often reduce performances, and there's nothing you can do except modifying your db schema. The simplest solution would be to have a table that precomputes data and reduces the work for your db engine. This table can be fulfilled with stored procedures triggered when you modify the data on the implied tables, or you can also modify the table values from your php application.