mysql query become slow when data reached upto 100000 - php

I build a query a month ago on a website. It was working fine. But after a month I was informed that the website become very slow to load the page.
When I search for the problem, I found that my query is executing very slow to fetch the data from mysql database. Then I check for the database and found that the 4 tables which I was using by joins, have around 216850, 167634, 64000, 931 rows respectively.
I have already have indexed that tables. So, where I'm lacking. Please help guys.
[Edit]
Table1: user_alert
Records: 216850
DB Type: InnoDB
Indexes: id(primary)
Table2: orders
Records: 167634
DB Type: InnoDB
Indexes: id(primary), order_id, customer_id
Table3: user_registration
Records: 64000 around
DB Type: InnoDB
Indexes: id(primary), email_address
Table4: cities
Records: 931
DB Type: InnoDB
Indexes: id(primary)
Query:
SELECT uas.alert_id, uas.user_id, uas.status, ur.first_name, ur.last_name, ur.email_address, o.order_id,
CASE WHEN ct.city_name IS NULL THEN uas.city_name ELSE ct.city_name END AS city_name
FROM `user_alert` uas
LEFT JOIN orders o ON o.customer_id = uas.user_id
LEFT JOIN user_registration ur ON ur.id = uas.user_id
LEFT JOIN `cities` ct ON ct.city_id = uas.city_id
WHERE uas.status = '1'
GROUP BY uas.user_id
ORDER BY uas.create_date DESC

GROUP BY is used to aggregate values up. For example if you wanted the count of orders by a user you could use COUNT(o.order_id).....GROUP BY uas.user_id. There are multiple orders for each user, but the aggregate function is just counting them here. However if you just select o.order_id when you have a GROUP BY uas.user_id it doesn't know which of the possibly many order_id values to return for that user id.
In this case it possibly doesn't matter as it looks like the order table is the only one where there is multiple rows per use. If you want the latest one you could just use MAX(o.order_id) (assuming that the order_id is assigned is order). But if you wanted the order value it becomes more difficult.
SELECT uas.alert_id, uas.user_id, uas.status, ur.first_name, ur.last_name, ur.email_address, MAX(o.order_id) AS LatestOrderId,
IFNULL(ct.city_name, uas.city_name) AS city_name
FROM `user_alert` uas
LEFT JOIN orders o ON o.customer_id = uas.user_id
LEFT JOIN user_registration ur ON ur.id = uas.user_id
LEFT JOIN `cities` ct ON ct.city_id = uas.city_id
WHERE uas.status = '1'
GROUP BY uas.user_id
ORDER BY uas.create_date DESC
If you wanted the (say) value of the latest order then it becomes more difficult.
SELECT uas.alert_id, uas.user_id, uas.status, ur.first_name, ur.last_name, ur.email_address, Sub1.MaxOrderId AS LatestOrderId, o.order_value
IFNULL(ct.city_name, uas.city_name) AS city_name
FROM `user_alert` uas
LEFT JOIN (SELECT customer_id, MAX(order_id) AS MaxOrderId FROM orders GROUP BY customer_id) Sub1 ON Sub1.customer_id = uas.user_id
LEFT OUTER JOIN orders o ON o.customer_id = Sub1.user_id AND o.order_id = Sub1.MaxOrderId
LEFT JOIN user_registration ur ON ur.id = uas.user_id
LEFT JOIN `cities` ct ON ct.city_id = uas.city_id
WHERE uas.status = '1'
ORDER BY uas.create_date DESC
Or doing a bit of a fiddle based on GROUP_CONCAT
SELECT uas.alert_id, uas.user_id, uas.status, ur.first_name, ur.last_name, ur.email_address,
SUBSTRING_INDEX(GROUP_CONCAT(o.order_id ORDER BY o.order_id DESC), ',', 1) AS LatestOrderId,
SUBSTRING_INDEX(GROUP_CONCAT(o.order_value ORDER BY o.order_id DESC), ',', 1) AS LatestOrderValue,
IFNULL(ct.city_name, uas.city_name) AS city_name
FROM `user_alert` uas
LEFT OUTER JOIN orders o ON o.customer_id = uas.user_id AND o.order_id = Sub1.MaxOrderId
LEFT JOIN user_registration ur ON ur.id = uas.user_id
LEFT JOIN `cities` ct ON ct.city_id = uas.city_id
WHERE uas.status = '1'
GROUP BY uas.user_id
ORDER BY uas.create_date DESC

Related

Mysql row number with order by very slow

I have this query
select courses.id, y.qs_world, courses.name_en as name,
courses.description_en as description,
source_link, courses.slug, fee, duration, courses.university_id, college_id,
study_level_id, application_fee, courses.currency_id
from courses
left join university_ranks as y on courses.university_id = y.university_id
and y.year = '2021'
left join universities on courses.university_id = universities.id
left join countries on countries.id = universities.country_id where courses.status = 1
order by ROW_NUMBER() OVER (PARTITION BY countries.id ORDER BY courses.id)
This query is taking too long to execute, but it is working well if I remove the last row.
I used indexing but nothing different.
The EXPLAIN notes are to Using temporary,Using filesort but I want to improve the query without using temporary or filesort
How can I achieve this?
UPDATE:
I tried this query but same speed
SELECT * FROM (
SELECT
`courses`.`id`,`courses`.`status`, `y`.`qs_world`, `courses`.`name_en` as `name`, `courses`.`description_en` as `description`,
`source_link`, `courses`.`slug`, `fee`, `duration`, `courses`.`university_id`, `college_id`,
`study_level_id`, `application_fee`, `courses`.`currency_id`, `countries`.`id` as country_id
FROM
courses
left join `university_ranks` as `y` on `courses`.`university_id` = `y`.`university_id`
and `y`.`year` = '2021'
left join `universities` on `courses`.`university_id` = `universities`.`id`
left join `countries` on `countries`.`id` = `universities`.`country_id`
) UserCourse where status = 1
order by ROW_NUMBER() OVER (PARTITION BY country_id ORDER BY id)
countries.id as country_id --> universities.country_id
then remove
left join `countries` ON `countries`.`id` = `universities`.`country_id`
Move where status = 1 into the inner query.
It seems like
order by ROW_NUMBER() OVER (PARTITION BY country_id ORDER BY id)
could be replaced by
ORDER BY country_id, id
Get rid of the outer query
Don't say LEFT unless the 'right' table might have a missing row. (It confuses the reader as to your intent.)

EXISTS query optimization on mysql query

I have a big data problem with MySQL.
I have:
a users table with 59033 rows, and
a user_notes table with 8753 rows.
But when I search which users have user note in some dates.
My query like this :
SELECT u.*, rep.name as rep_name FROM users as u
LEFT JOIN users as rep on rep.id = u.add_user
LEFT JOIN authorization on authorization.id = u.authorization
LEFT JOIN user_situation_list on user_situation_list.user_situation_id = u.user_situation
WHERE
EXISTS(
select * from user_notes
where user_notes.note_user_id = u.id AND user_notes.create_date
BETWEEN "2017-10-20" AND "2017-10-22"
)
ORDER BY u.lp_modify_date DESC, u.id DESC
Turn it around -- find the ids first; deal with the joins later.
SELECT u.*,
( SELECT rep.name
FROM users AS rep
WHERE rep.id = u.add_user ) AS rep_name
FROM (
SELECT DISTINCT note_user_id
FROM user_notes
WHERE create_date >= "2017-10-20"
AND create_date < "2017-10-20" + INTERVAL 3 DAY
) AS un
JOIN users AS u ON u.id = un.note_user_id
ORDER BY lp_modify_date DESC, id DESC
Notes
No GROUP BY needed;
2 tables seem to be unused; I removed them;
I changed the date range;
User notes needs INDEX(create_date, note_user_id);
Notice how I turned a LEFT JOIN into a subquery in the SELECT list.
If there can be multiple rep_names, then the original query is "wrong" in that the GROUP BY will pick a random name. My Answer can be 'fixed' by changing rep.name to one of these:
MAX(rep.name) -- deliver only one; arbitrarily the max
GROUP_CONCAT(rep.name) -- deliver a commalist of names
Rewriting your query to use a JOIN rather than an EXISTS check in the where should speed it up. If you then group the results by the user.id it should give you the same result:
SELECT u.*, rep.name as rep_name FROM users as u
LEFT JOIN users as rep on rep.id = u.add_user
LEFT JOIN authorization on authorization.id = u.authorization
LEFT JOIN user_situation_list on user_situation_list.user_situation_id = u.user_situation
JOIN user_notes AS un
ON un.note_user_id
AND un.create_date BETWEEN "2017-10-20" AND "2017-10-22"
GROUP BY u.id
ORDER BY u.lp_modify_date DESC, u.id DESC

How to sum of two different table with two different groupby

I have two tables called work and stuff both tables have same fields -
company, quality, quantity
I need the sum of all the quantity of work with group by company and quality and join the table with sum of all the quantity of stuff with group by company and quality
I didn't get the expected result.
SQLfiddel
http://sqlfiddle.com/#!9/eea577/6
working query
select st.company,st.quality,st.quantitys - ct.quantitys as balance from
(select company,quality,sum(quantity) as quantitys from stuff
group by quality,company) as st join (select company,quality,
sum(quantity) as quantitys from work group by quality,company)
as ct on `ct`.`company` = `st`.`company` and ct.quality = st.quality group by quality,company
u have missed the quality condition in the join
Try the following:
SELECT tableA.ID, tableA.`Year`, tableA.`Month`,
tableA.`Type`, tableA.instrument,
tableA.totalAmount, tableB.totalInstrument
FROM
(
SELECT a.ID, a.`Year`, a.`Month`,
b.`Type`, b.instrument,
SUM(b.`amount`) totalAmount
FROM `date` a
INNER JOIN `transactions` b
ON a.ID = b.id
GROUP BY b.`Type
) tableA
INNER JOIN
(
SELECT a.ID, a.`Year`, a.`Month`,
b.`Type`, b.instrument,
SUM(b.`instrument`) totalInstrument
FROM `date` a
INNER JOIN `transactions` b
ON a.ID = b.id
GROUP BY a.`Year`, a.`Month`
) tableB ON tableA.ID = tableB.ID AND
tableA.`Year` = tableB.`Year` AND
tableA.`Month` = tableB.`Month`

MySQL - Having problems with COUNT() in query with GROUP BY

I have a pretty big query which is used by an ajax call to return and also sort active items. From my understanding sub queries should be avoided where possible and since this query will be called very often, I would like to do just that.
At the moment everything is fine except for the COUNT(b.bic) AS bids. If there are two(2) bids the query returns four(4), if there are 4, it returns 8, and so on. I've tried grouping by other columns ... but no luck.
Some of the tables. I hope each of the column names are pretty self explanatory:
countries_ship - each item can be shipped to multiple countries so item_id can be duplicate.
id item_id country_id ship_cost
countries
id country_code country_name
item_expire - not sure if item_expire should have it's own table.
id item_id exp_date
bids - Just as countries_ship, item_id can be duplicate. This is where the bids are stored.
id item_id user_id bid previous_bid bid_date
The query:
$q = $this->db->mysqli->prepare("
SELECT c.ship_cost,
c.item_id,
co.country_name,
co.id AS co_id,
i.id,
i.user_id,
i.item_start,
i.item_title,
i.item_number,
i.item_year,
i.item_publisher,
i.item_condition,
i.item_description,
i.item_location,
e.exp_date AS exp_date,
i.active,
CAST(u.fb_id AS CHAR(50)) AS fb_id,
u.user_pic,
MAX(b.bid) AS maxbid,
COUNT(b.bid) AS bids,
p.publisher_name,
t.tag_name
FROM countries_ship c
JOIN items i
ON c.item_id = i.id
JOIN item_expire e
ON c.item_id = e.item_id
JOIN users u
ON i.user_id = u.id
LEFT JOIN bids b
ON i.id = b.item_id
LEFT JOIN publishers p
ON i.item_publisher = p.id
LEFT JOIN tags_rel tr
ON c.item_id = tr.item_id
JOIN tags t
ON t.id = tr.tag_id
LEFT JOIN countries co
ON i.item_location = co.id
WHERE ".$where."
GROUP BY c.item_id ORDER BY ".$order." ".$limit."");
You may try
COUNT(distinct b.bic) AS bids
This will ignore duplicates due to joins

SQL query problem

I've got reporting of a user's score everytime it happens. Now I want to show the best score a user has had. The table set up is like this:
Player(id, name)
PlayerHasAchievement(id, playerId,
achievementId)
Achievement(id, type, amount, time)
This is what I have right now:
$query = "SELECT MAX(ach.amount) as amount, p.username, ach.time
FROM achievement as ach
INNER JOIN playerHasAchievement as playAch ON ach.id = playAch.id
INNER JOIN player as p ON p.userId = playAch.userid
WHERE ach.type = 2
GROUP BY amount
ORDER by `amount` DESC
LIMIT $amount";
I tried to select it distinctly but it didn't work. I'm stumped, it's supposed to be so easy! Thanks for reading, I'll be grateful for any help!
The problem is the the ach.time you are getting is not the same row as the MAX(amount). Join another subquery to get the MAX(amount) first.
Note: In the table definitions you posted, playerHasAchievement has a field playerId not userId
SELECT MAX(ach.amount) as amount, p.username, MAX(ach.time) MaxTime
FROM achievement as ach
INNER JOIN playerHasAchievement as playAch ON ach.id = playAch.id
INNER JOIN player as p ON p.userId = playAch.playerId
INNER JOIN (
SELECT playAch.playerId, MAX(ach.amount) as MaxAmount
FROM achievement as ach
INNER JOIN playerHasAchievement as playAch ON ach.id = playAch.id
WHERE ach.type = 2
GROUP BY playAch.playerId
) g ON p.playerId = g.playerId AND ach.amount = g.MaxAmount
WHERE ach.type = 2
GROUP BY p.playerId
ORDER by `amount` DESC
LIMIT $amount";
The reason why we group the outer query, is to avoid ties - say a player had the same score twice.
In your join on line 3 don't you really want
INNER JOIN playerHasAchievement as playAch ON ach.id = playAch.achievementId
and others are correct, you need to group by your non aggregate columns, not the aggregate one.
Assuming your db layout is as specified in the question here is the query I would use.
SELECT ach.amount, p.Name, ach.time
FROM achievement as ach
JOIN playerHasAchievement as playAch ON ach.id=playAch.achievementId
JOIN player AS p ON p.id = playAch.playerId
WHERE ach.type = 2
AND ach.amount = (SELECT MAX(ach.amount)
FROM achievement as ach
JOIN playerHasAchievement as playAch ON ach.id=playAch.achievementId
JOIN player AS p ON p.id = playAch.playerId
WHERE ach.type = 2)
GROUP BY ach.amount
ORDER by ach.time
taking the first result (in case there are multiples of the same score) will give you the high score and the lowest time.
Hope that helps!
You are not using group by appropriately, as you are only grouping by amount.
What about the user name and the time?

Categories