I'm working on a project which have a large set of data. The data are stored in a MySql Db.
I want to fetch records with pagination from few tables. One of the table is having over 2 millions of records which is causing the page to freeze for a very long time. Also sometimes the page is not able to load at all.
The query I'm using to fetch the records :
SELECT
EN.`MRN`,
P.`FNAME`,
P.`LNAME`,
P.`MI`,
P.`SSC`,
sum(EN.`AMOUNT`) AS `TOTAL_AMOUNT`
FROM `table_1` AS EN
INNER JOIN `table_2` AS P ON EN.`MRN` = P.`MRN`
GROUP BY EN.`MRN`,P.`FNAME`, P.`LNAME`,P.`MI`,P.`SSC`
HAVING sum(EN.`AMOUNT`) > 0
ORDER BY P.`LNAME`
By this query I'm getting the total number of records for the pagination to work. Then I again run this query to get the actual records :
SELECT
EN.`MRN`,
P.`FNAME`,
P.`LNAME`,
P.`MI`,
P.`SSC`,
sum(EN.`AMOUNT`) AS `TOTAL_AMOUNT`
FROM `table_1` AS EN
INNER JOIN `table_2` AS P ON EN.`MRN` = P.`MRN`
GROUP BY EN.`MRN`,P.`FNAME`, P.`LNAME`,P.`MI`,P.`SSC`
HAVING sum(EN.`AMOUNT`) > 0
ORDER BY P.`LNAME`
LIMIT 0, 100
How can I make this query to work faster. Because it takes a very long time execute the query for the first time to get total number of records.
It is better to separate total_amount from the query because that is the only value involving all the records. You can call this when you load the page. I assume that all your records in table_1 is valid.
SELECT sum(EN.`AMOUNT`) AS `TOTAL_AMOUNT` FROM `table_1` AS EN
HAVING sum(EN.`AMOUNT`) > 0
Then get the query result every time when you flip the page. This should only return 10 records starting from record 0.
SELECT
EN.`MRN`,
P.`FNAME`,
P.`LNAME`,
P.`MI`,
P.`SSC`
FROM `table_1` AS EN
INNER JOIN `table_2` AS P ON EN.`MRN` = P.`MRN`
GROUP BY EN.`MRN`,P.`FNAME`, P.`LNAME`,P.`MI`,P.`SSC`
HAVING sum(EN.`AMOUNT`) > 0
ORDER BY P.`LNAME`
LIMIT 10 OFFSET 0
Hope this helps.
There are a few things you can do to make this faster:
Use EXPLAIN to make sure your indices are set correctly / set the correct indices.
Only execute the complicated query once using SQL_CALC_FOUND_ROWS.
Use the TOTAL_AMOUNT alias in your HAVING statement to avoid doing the calculation twice.
MySQL will do some optimizations and caching itself so they might not all have the same impact.
Related
I write a join query for two tables, with 4000 records each, to fetch the data using php. I execute the query in mysql and it takes 32 seconds.
Here is the query
SELECT Distinct a.*,b.*
FROM OrderCalculation a right join crm_order b on a.orderid = b.orderno
order by b.orderno desc LIMIT 20 OFFSET 0
Can anybody please improve this query and reduce the execution time
Thanks
I can offer the following index suggestion, which might improve your query performance:
CREATE INDEX idx ON crm_order (orderno, col1, col2, ...);
I would only suggest this if you have no more than say 4-5 other columns in the crm_order table to cover. If you have more than that, it might make sense to not use this index.
If it helps, the index is working by letting MySQL rapidly lookup each orderno value from the OrderCalculation table. This would also assume that MySQL would do a full table scan of the OrderCalculation table. You might want to first run EXPLAIN on your current query to see where you stand.
As your join is using orderid and orderno columns of OrderCalculation and crm_order tables respectively, you can add index on these two columns and measure the performance.
Here's MySQL documentation on how to create index.
Just for reference, we would normally write that query this way:
SELECT c.olumns
, y.ou
, a.ctually
, w.ant
FROM crm_order a
LEFT
JOIN OrderCalculation b
ON b.orderid = a.orderno
ORDER
BY b.orderno DESC
LIMIT 0,20
SELECT
r.id randevu_id,
r.id,m.ad,m.soyad,r.musteri_id,m.telefon,
r.tarih,r.hizmet_id,
IFNULL((SELECT MAX(o.tarih)
FROM odemeler o WHERE o.sil=0 AND o.randevu_id=r.id),"0000-00-00") sonodemetarih,
r.tutar borc
FROM randevular r
LEFT JOIN musteriler m ON m.id = r.musteri_id
WHERE r.sil = 0 AND r.randevu_durumu != 2
in PhpMyAdmin it tooks 0.6 seconds but in php with PDO it tooks 10 minutes. And when i delete this part
IFNULL((SELECT MAX(o.tarih)
FROM odemeler o WHERE o.sil=0 AND o.randevu_id=r.id),"0000-00-00") sonodemetarih
it tooks 1 seconds in php
Why could it be? can you help me, Thanks.
Edit :
you can only focus this query,
SELECT
(SELECT count(1) FROM payments p WHERE p.appointment_id=ap.id AND p.sil=0 )
as paymentscount
FROM appointments ap
in PhpMyAdmin it tooks 0.6 seconds but in php with PDO it tooks 10 minutes.
What accounts for the high performance in myAdmin? Probably the query cache. When testing queries for performance use SELECT SQL_NO_CACHE ... or MySQL will remember the previous query result and give it to you again.
What accounts for the slow performance of the query? It has a dependent subquery, and MySQL is probably repeating that subquery for each row of the main query.
Try refactoring your query to avoid the dependent subquery, and instead to join to an independent summary subquery.
Your subquery is this:
SELECT COUNT(*) payment_count, appointment_id
FROM payments
WHERE sil=0
GROUP BY appointment_id
It gives a little table of payment count by appointment.
Join that to the rest of your query:
SELECT ap.*, pc.payment_count
FROM appointments ap
LEFT JOIN (
SELECT COUNT(*) payment_count, appointment_id
FROM payments
WHERE sil=0
GROUP BY appointment_id
) pc ON ap.id = pc.appointment_id
This is faster because it only has to generate the subquery table once.
The knack you need here is generating the summary you need as a subquery, and using it as a virtual table.
I am generating a CSV of how many orders I have every month for the last 12 months per item. At the moment I am retrieving the items and using a foreach loop to submit the following query for each item, in order to retrieve the total times it has been ordered in each month over the past 12 months. However, the query takes just over 3 seconds, so when the loop goes a couple of thousand times, it causes the MySQL server to go away.
How could I optimise this query? Should the load on the database be reduced if I use a sub query instead?
Here is the query(which was adapted from here):
SELECT order_item_variant_alias_id, DATE(DATE_FORMAT(order_progress_time, '%Y-%m-01')) AS `trueMonth`, COUNT(*) AS count
FROM tbl_order_progress
JOIN tbl_order_items ON order_progress_order_id = order_item_order_id
JOIN tbl_product_variant_aliases ON order_item_variant_alias_id = product_variant_alias_id
JOIN tbl_product_variants ON product_variant_id = product_variant_alias_variant_id
GROUP BY product_variant_alias_id, DATE(DATE_FORMAT(order_progress_time, '%Y-%m-01'))
HAVING order_item_variant_alias_id = 1
Why are you running a separate query for each item? Just drop the having clause and put all items in at the same time.
If, for some reason, you do need to do one item at a time, switch the logic to a where clause rather than a having clause. The having clause will aggregate all items and then filter down to the one you want. MySQL should be faster if you reduce the data first -- using where:
SELECT order_item_variant_alias_id,
DATE(DATE_FORMAT(order_progress_time, '%Y-%m-01')) AS `trueMonth`,
COUNT(*) AS count
FROM tbl_order_progress JOIN
tbl_order_items
ON order_progress_order_id = order_item_order_id JOIN
tbl_product_variant_aliases
ON order_item_variant_alias_id = product_variant_alias_id JOIN
tbl_product_variants
ON product_variant_id = product_variant_alias_variant_id
WHERE order_item_variant_alias_id = 1
GROUP BY product_variant_alias_id,
DATE(DATE_FORMAT(order_progress_time, '%Y-%m-01'));
I have 100 records in a sql table
can I get the total number of records in single query with limits applied
like ..
SELECT * , COUNT(*) as records form table offset 0 limit 10.
So the records will have value 100.
Is possible some how ?
Use a correlated subquery:
SELECT * , (SELECT COUNT(*)
from table) as records
form table
offset 0 limit 10;
But it would be better to do only one separate query for this count, then from your front end application display it the way you want, instead of doing it 100 times.
I have to modify an existing PHP script so that it displays the total number of matches in the database before it shows the results for the current page. The query is:
SELECT products.features, products.make_id, products.model, products.price,
products.id, brands.name AS brandname, brands.id AS brandid
FROM products
LEFT JOIN brands ON products.make_id=brands.id
WHERE products.active=1 AND brands.active=1 AND (products.category=22)
ORDER BY id DESC LIMIT 0,12
How do I get the number of results? mysql_num_rows() obviously returns 12 but I don't know how to use count in this query, everything I try returns some error.
I could just execute the query once without the limit (clause?) but that seems a bit inefficient.
Thanks.
Use the MySQL function FOUND_ROWS():
A SELECT statement may include a LIMIT
clause to restrict the number of rows
the server returns to the client. In
some cases, it is desirable to know
how many rows the statement would have
returned without the LIMIT, but
without running the statement again.
To obtain this row count, include a
SQL_CALC_FOUND_ROWS option in the
SELECT statement, and then invoke
FOUND_ROWS() afterward:
so run the two following queries in PHP, the first statement gives the first 12 rows, the second one gives the total of rows:
$sql1 = "SELECT SQL_CALC_FOUND_ROWS
products.features
, products.make_id
, products.model
, products.price
, products.id
, brands.name AS brandname
, brands.id AS brandid
FROM products
LEFT JOIN brands ON products.make_id=brands.id
WHERE (products.active=1) AND (brands.active=1) AND (products.category=22)
ORDER BY id DESC LIMIT 0,12"
$sql2 = "SELECT FOUND_ROWS()"
This would return you the total number of rows the current select matches. The result will be one row, with one field and it will hold the total
SELECT COUNT(*) AS total
FROM products
LEFT JOIN brands ON products.make_id=brands.id
WHERE products.active=1 AND brands.active=1 AND products.category=22
You will have to run this as a separate query.
<?php
$q=mysql_query('above query');
$result=mysql_fetch_array($q);
echo $result['total']; // will print out the desired number of rows
You were probably trying to get the total and the limit in the same select and while that might be possible with a union, but since you were worrying about inefficiency before, this is possible worse.
Since in your OP you hint that this is for pagination. Let me tell you, LIMIT m,n may not be as fast as it sounds. Learn how to improve it and read more about Efficient Pagination Using MySQL