PHP - SQL optimization min/max too slow - php

I'm having some problems with a query that finds the next ID of an orders with certain filters on it - Like it should be from a specific city, etc.
Currently it's used for a function, where it'll either spit out the previous or the next ID based on the current order. So it can either be min(id) or max(id), where max(id) is obviously faster, since it has to go through less rows.
The query is working just fine, but it's rather slow, since it's going through 123953 rows to find the ID. Is there any way I could optimize this?
Function example:
SELECT $minmax(orders.orders_id) AS new_id FROM orders LEFT JOIN orders_status ON orders.orders_status = orders_status.orders_status_id $where_join WHERE orders_status.language_id = '$languages_id' AND orders.orders_date_finished != '1900-01-01 00:00:00' AND orders.orders_id $largersmaller $ordersid $where;
Live example
SELECT min(orders.orders_id)
FROM orders
LEFT JOIN orders_status ON orders.orders_status = orders_status.orders_status_id
WHERE orders_status.language_id = '4'
AND orders.orders_date_finished != '1900-01-01 00:00:00'
AND orders.orders_id < 4868771
LIMIT 1

so concluding:
SELECT orders.orders_id
FROM orders
JOIN orders_status ON orders.orders_status = orders_status.orders_status_id
WHERE orders_status.language_id = '4'
AND orders.orders_date_finished != '1900-01-01 00:00:00'
AND orders.orders_id < 4868771
ORDER BY orders.orders_id ASC
LIMIT 1
Extra:
to get the MAX value, use DESC where ASC is now.
And looking at your question: be sure to escape the values like $language_id etcetera. I suppose they could come from some html form?
(or use prepared statements)

Related

Single query that allows alias with it's own limit

I would like to better optimize my code. I'd like to have a single query that allows an alias name to have it's own limit and also include a result with no limit.
Currently I'm using two queries like this:
// ALL TIME //
$mikep = mysqli_query($link, "SELECT tasks.EID, reports.how_did_gig_go FROM tasks INNER JOIN reports ON tasks.EID=reports.eid WHERE `priority` IS NOT NULL AND `partners_name` IS NOT NULL AND mike IS NOT NULL GROUP BY EID ORDER BY tasks.show_date DESC;");
$num_rows_mikep = mysqli_num_rows($mikep);
$rating_sum_mikep = 0;
while ($row = mysqli_fetch_assoc($mikep)) {
$rating_mikep = $row['how_did_gig_go'];
$rating_sum_mikep += $rating_mikep;
}
$average_mikep = $rating_sum_mikep/$num_rows_mikep;
// AND NOW WITH A LIMIT 10 //
$mikep_limit = mysqli_query($link, "SELECT tasks.EID, reports.how_did_gig_go FROM tasks INNER JOIN reports ON tasks.EID=reports.eid WHERE `priority` IS NOT NULL AND `partners_name` IS NOT NULL AND mike IS NOT NULL GROUP BY EID ORDER BY tasks.show_date DESC LIMIT 10;");
$num_rows_mikep_limit = mysqli_num_rows($mikep_limit);
$rating_sum_mikep_limit = 0;
while ($row = mysqli_fetch_assoc($mikep_limit)) {
$rating_mikep_limit = $row['how_did_gig_go'];
$rating_sum_mikep_limit += $rating_mikep_limit;
}
$average_mikep_limit = $rating_sum_mikep_limit/$num_rows_mikep_limit;
This allows me to show an all-time average and also an average over the last 10 reviews. Is it really necessary for me to set up two queries?
Also, I understand I could get the sum in the query, but not all the values are numbers, so I've actually converted them in PHP, but left out that code in order to try and simplify what is displayed in the code.
All-time average and average over the last 10 reviews
In the best case scenario, where your column how_did_gig_go was 100% numeric, a single query like this could work like so:
SELECT
AVG(how_did_gig_go) AS avg_how_did_gig_go
, SUM(CASE
WHEN rn <= 10 THEN how_did_gig_go
ELSE 0
END) / 10 AS latest10_avg
FROM (
SELECT
#num + 1 AS rn
, tasks.show_date
, reports.how_did_gig_go
FROM tasks
INNER JOIN reports ON tasks.EID = reports.eid
CROSS JOIN ( SELECT #num := 0 AS n ) AS v
WHERE priority IS NOT NULL
AND partners_name IS NOT NULL
AND mike IS NOT NULL
ORDER BY tasks.show_date DESC
) AS d
But; Unless all the "numbers" are in fact numeric you are doomed to sending every row back from the server for php to process unless you can clean-up the data in MySQL somehow.
You might avoid sending all that data twice if you establish a way for your php to use only the top 10 from the whole list. There are probably way of doing that in PHP.
If you wanted assistance in SQL to do that, then maybe having 2 columns would help, it would reduce the number of table scans.
SELECT
EID
, how_did_gig_go
, CASE
WHEN rn <= 10 THEN how_did_gig_go
ELSE 0
END AS latest10_how_did_gig_go
FROM (
SELECT
#num + 1 AS rn
, tasks.EID
, reports.how_did_gig_go
FROM tasks
INNER JOIN reports ON tasks.EID = reports.eid
CROSS JOIN ( SELECT #num := 0 AS n ) AS v
WHERE priority IS NOT NULL
AND partners_name IS NOT NULL
AND mike IS NOT NULL
ORDER BY tasks.show_date DESC
) AS d
In future (MySQL 8.x) ROW_NUMBER() OVER(order by tasks.show_date DESC) would be a better method than the "roll your own" row numbering (using #num+1) shown before.

Sorting data from MySQL by SUM with Grouping By - not working properly

I have a problem with mysql - i'm kind new to it, byt looking to improve my skills :)
Have a code like this:
if($where > 0) $query = mysql_query("SELECT img.*, user.user as owner_name, cat.name as cat_name FROM tentego_img AS img LEFT JOIN tablicacms_users AS user ON user.id = img.owner LEFT JOIN tentego_img_cat AS cat ON cat.id = img.cat WHERE img.`is_waiting` LIKE ".$where.$cat." INNER JOIN tentego_img_vote ON tentego_img.id = tentego_img_vote.object_id GROUP BY tentego_img_vote.object_id ORDER BY SUM ( (CASE WHEN tentego_img_vote.vote = '0' THEN '-1' ELSE '1' END) ) DESC LIMIT ".$page.",".$objPerPage);
I need to make sorting by number of votes, sorted descending.
Still it makes results sorted by it own way.
In table I have rows:
ID - vote id for table purpose
object_id- id of object joined with another table to show results.
User ID - user id
Vote - where values are 0 for dislike and 1 for like (so -1 for 0, and +1 for 1)
So, as I understand i need to sum up all records for each of unique object_id, then sort by sum of vote values of each.
This code worked before my script provider decide to upgrade it, so right now i dont know how to fix it :(

reduce left join query excution time

I'm developing an sql query that can join two table and it returns some results.
I have 2 tables in first table i save my order and in another table save my like information .
I want to show to user picture from order table that, user doesn't like picture yet . I use this query
SELECT amg_order.*
FROM amg_order
LEFT OUTER JOIN amg_like ON amg_like.order_id=amg_order.order_id
AND amg_like.user_id=:user_id
WHERE amg_order.status = '1'
AND amg_order.user_id != :user_id
AND (amg_like.user_id != :user_id || amg_like.user_id is null)
ORDER BY amg_order.likeType DESC, RAND()
This query return correct result but when like information be over 15000 time to execution this query has been 6 seconds .
Does anyone has any idea to reduce this time ?
I'm sorry my English is so bad :)
You can try following query. This will of course reduce some of your execution time. You can specify fields name instead of * sign in your select statement.
Here is updated query:
SELECT amg_order.* FROM amg_order
LEFT JOIN amg_like ON amg_order.order_id = amg_like.order_id
WHERE amg_order.status= '1' AND amg_order.user_id != :user_id AND (amg_like.user_id != :user_id || amg_like.user_id is null)
ORDER BY amg_order.likeType DESC LIMIT 10;

Make SQL query faster

Can anyone tell me how to make this query faster?
$session_id = '000000000015';
$start = 0;
$finish = 30;
try {
$stmt = $conn->prepare("SELECT TOPUSERS.ID, TOPUSERS.USERNAME, TOPUSERS.NAME, TOPUSERS.NAME2, TOPUSERS.PHOTO, TOPUSERS.FB_USERID, TOPUSERS.IMAGE_TYPE, TOPUSERS.TW_USERID, TOPUSERS.TW_PHOTO,
COALESCE((SELECT COUNT(USERS_BUCKETS.ID) FROM USERS_BUCKETS WHERE USERS_BUCKETS.USERID=TOPUSERS.ID),0) AS NUM_ALL,
COALESCE((SELECT SUM(CASE WHEN USERS_BUCKETS.STATUS='Completed' THEN 1 ELSE 0 END) FROM USERS_BUCKETS WHERE USERS_BUCKETS.USERID=TOPUSERS.ID),0) AS NUM_DONE,
COALESCE((SELECT COUNT(USERS_LIKES.ID) FROM USERS_LIKES WHERE USERS_LIKES.USERID=TOPUSERS.ID),0) AS NUM_LIKES,
(SELECT USERS_BUCKETS.BUCKETID FROM USERS_BUCKETS WHERE USERS_BUCKETS.USERID=TOPUSERS.ID ORDER BY USERS_BUCKETS.DATE_MODIFIED DESC LIMIT 1) AS RECENT_BUCKET,
(SELECT BUCKETS_NEW.BUCKET_NAME FROM BUCKETS_NEW WHERE BUCKETS_NEW.ID=RECENT_BUCKET) AS REC,
COALESCE((SELECT COUNT(ID) FROM FOLLOW WHERE FOLLOW.USER_ID=TOPUSERS.ID),0) AS FOLLOWING,
COALESCE((SELECT COUNT(ID) FROM FOLLOW WHERE FOLLOW.FOLLOW_ID=TOPUSERS.ID),0) AS FOLLOWERS,
(SELECT IF(TOPUSERS.NAME = '',0,1) + IF(TOPUSERS.BIO = '',0,1) + IF(TOPUSERS.LOCATION = '',0,1) + IF(TOPUSERS.BIRTHDAY = '0000-00-00',0,1) + IF(TOPUSERS.GENDER = '',0,1)) as COMPLETENESS,
CASE WHEN ? IN (SELECT USER_ID FROM FOLLOW WHERE FOLLOW_ID = TOPUSERS.ID) THEN 'Yes' ELSE 'No' END AS DO_I_FOLLOW_HIM
FROM TOPUSERS
LEFT JOIN FOLLOW ON TOPUSERS.ID = FOLLOW.FOLLOW_ID
LEFT JOIN USERS_BUCKETS ON USERS_BUCKETS.USERID=TOPUSERS.ID
LEFT JOIN BUCKETS_NEW ON BUCKETS_NEW.ID=USERS_BUCKETS.BUCKETID
WHERE NOT TOPUSERS.ID = ?
GROUP BY TOPUSERS.ID ORDER BY TOPUSERS.RANDOM, TOPUSERS.USERNAME LIMIT $start, $finish");
When I run this in a browser it takes about 7 seconds to load. Without a few lines (the COALESCE in the middle, the two SELECTS above and the line below them) the time is reduced to 3-4 seconds.
The result of the query is a list of people with names, profile picture and some data.
TL,DR: you need to rewrite the query.
You need to rewrite your query to make it more efficient. I had to rewrite a similar query at work last week and here is what I have done.
The structure of your query should look like this to be efficient:
select ...
...
from ...
join ...
where ...
what you have now is something like:
select ...
inner select
inner select
from ...
join ...
where ...
That's the inner selects that kill your query. You need to find a way to move the inner select into the from section. Especially that you already query the tables.
What you need to understand is that your inner selects run for every records you have. So if you have 10 records, it would be alright (speed wise). But with hundred or thousand of records, it would be very slow.
If you want more information on your query run it with the explain keyword in from of it.

Need Help optimizing a complex MySQL query

I have this query below. There are 4 main tables involved: tblOrder, tblItems, tblOrder_archive, tblItem_archive. Orders and Items get moved over to the archived versions of the tables after a few months as not to slow down the main table queries. (sales and traffic is REALLY HIGH). So to get sales figures, i select what i need from each set of tables (archive and non archive).. union them.. do a group by on the union.. then do some math on the result.
Problem is that with any significant amount of rows (the order time span).. it will take so long for the query to run that it times out. I have added all the keys I can think of and still running super slow.
Is there more I can do to make this run faster? Can i write it differently? Can i use different indexes?
or should i write a script that gets the data from each table set first then does the math in the php script to combine them?
Thanks for the help.
SELECT
description_invoice
, supplier
, type
, sum(quantity) AS num_sold
, sum(quantity*wholesale) AS wholesale_price
, sum(quantity*price) AS retail_price
, sum(quantity*price) - sum(quantity*wholesale) AS profit
FROM (
SELECT
tblOrder.*
, tblItem.description_invoice
, tblItem.type
, tblItem.product_number
, tblItem.quantity
, tblItem.wholesale
, tblItem.price
, tblItem.supplier
FROM tblOrder USE KEY (finalized), tblItem
WHERE
tblItem.order_id = tblOrder.order_id
AND
finalized=1
AND
wholesale <> 0
AND (order_time >= 1251788400 AND order_time <= 1283669999)
UNION
SELECT
tblOrder_archive.*
, tblItem_archive.description_invoice
, tblItem_archive.type
, tblItem_archive.product_number
, tblItem_archive.quantity
, tblItem_archive.wholesale
, tblItem_archive.price
, tblItem_archive.supplier
FROM tblOrder_archive USE KEY (finalized), tblItem_archive
WHERE
tblItem_archive.order_id=tblOrder_archive.order_id
AND
finalized=1
AND
wholesale <> 0
AND (order_time >= 1251788400 AND order_time <= 1283669999)
) AS main_table
GROUP BY
description_invoice
, supplier,type
ORDER BY profit DESC;
Create indexes on the columns you are using in the WHERE clauses.
Remove the index hint: USE KEY (finalized). If it does anything at all it will probably just make it slower by causing MySQL to choose this key instead of a potentially better key.
Add a LIMIT to avoid fetching too many rows. Use paging if you want to see more rows.
Use UNION ALL instead of UNION. This will be faster because it doesn't check for duplicates and also you probably don't want to remove duplicates here anyway since this will affect the total.
Orders and Items get moved over to the archived versions of the tables after a few months as not to slow down the main table queries.
This is probably a bad idea. Instead you should index your data correctly so that the queries don't become significantly slower when you add more data. Or alternatively you could look at partitioning the table.
I re-wrote your query as:
SELECT COALESCE(x.description_invoice, y.description_invoice) AS description_invoice,
COALESCE(x.supplier, y.supplier) AS supplier,
COALESCE(x.type, y.type) AS type,
COALESCE(SUM(x.quantity), 0) + COALESCE(SUM(y.quantity), 0) as num_sold,
COALESCE(SUM(x.quantity * x.wholesale), 0) + COALESCE(SUM(y.quantity * y.wholesale), 0) AS wholesale_price,
COALESCE(SUM(x.quantity * x.price), 0) + COALESCE(SUM(y.quantity * y.price), 0) AS retail_price,
COALESCE(SUM(x.quantity * x.price), 0) - COALESCE(SUM(x.quantity * x.wholesale), 0) + COALESCE(SUM(y.quantity * y.price), 0) - COALESCE(SUM(y.quantity * y.wholesale), 0) as profit
FROM (SELECT o.order_id
FROM TBLORDER o
WHERE o.finalized = 1
AND o.order_time BETWEEN 1251788400
AND 1283669999
UNION ALL
SELECT oa.order_id
FROM TBLORDER_ARCHIVE oa
WHERE oa.finalized = 1
AND oa.order_time BETWEEN 1251788400
AND 1283669999) a
LEFT JOIN TBLITEM x ON x.order_id = a.order_id
AND x.wholesale != 0
LEFT JOIN TBLITEM_ARCHIVE y ON y.order_id = a.order_id
AND y.wholesale != 0
GROUP BY description_invoice, supplier, type
ORDER BY profit DESC
Your query had UNION, but I'd expect not to need duplicate removal from an archive table so I changed it to UNION ALL - which is faster, because it doesn't remove duplicates
For what you provided, you had SELECT ORDERS.* and SELECT ORDER_ARCHIVE.* but never used any of the columns.
The aggregation functions (SUM) were all on the TBLITEM table, which was unnecessarily within the derived table/inline view.
I omitted the USE KEY(finalized); you can re-add it if you like but I'd compare with and with out it - I'd suggest running ANALYZE TABLE occaissionally on both tables prior to running the query so the optimizer has relatively fresh statistics.
I don't see much value in an index on the finalized column, but I don't know your data or use - just this query. But based on this query, I'd index:
order_id
order_time
finalized
...as a covering index--a single index with three columns, in the order provided because order is important in a covering index.
I rewrote it as follows based on your help, and added the recommended covering index to both tblOrder and tblOrder archive and things seem to be much faster. But still i'm wondering if there something more to the way you wrote it.. but i would need to use tblItem_archive joined to tblOrder_archive as well.
SELECT
description_invoice
, supplier
, type
, sum(quantity) AS num_sold
, sum(quantity*wholesale) AS wholesale_price
, sum(quantity*price) AS retail_price
, sum(quantity*price) - sum(quantity*wholesale) AS profit
FROM (
SELECT
tblOrder.order_id
, tblItem.description_invoice
, tblItem.type
, tblItem.product_number
, tblItem.quantity
, tblItem.wholesale
, tblItem.price
, tblItem.supplier
FROM tblOrder, tblItem
WHERE
tblItem.order_id = tblOrder.order_id
AND
finalized=1
AND
wholesale <> 0
AND (order_time >= 1251788400 AND order_time <= 1283669999)
UNION ALL
SELECT
tblOrder_archive.order_id
, tblItem_archive.description_invoice
, tblItem_archive.type
, tblItem_archive.product_number
, tblItem_archive.quantity
, tblItem_archive.wholesale
, tblItem_archive.price
, tblItem_archive.supplier
FROM tblOrder_archive, tblItem_archive
WHERE
tblItem_archive.order_id=tblOrder_archive.order_id
AND
finalized=1
AND
wholesale <> 0
AND (order_time >= 1251788400 AND order_time <= 1283669999)
) AS main_table
GROUP BY
description_invoice
, supplier,type
ORDER BY profit DESC;

Categories