I am currently working on speeding up a website, that is returning 300,000+ rows from a query. While I don't think this is too much of a load on the DB server, this query is happening in a while loop depending on the number of 'galleries' a user has.
For example Joe has 10 galleries in his account. Each of those galleries has x number of images, which have x number of comments on those images. So the query that is currently being run...
SELECT count(*) as total
FROM galleryimage a
INNER JOIN imagecomments b ON a.id=b.imgId
WHERE a.galleryId='".$row['id']."'
AND b.note <> ''
...is looking through all the galleryimage table 334,000 rows and the imagecomments table 76,000 rows and returning the result on each gallery. The query run on a single gallery returns a result in about 578ms, but with many galleries, say 30-40 you could be looking at a page load time of 17+ secs. Any suggestions on how to deal with this issue?
I cannot change the DB architecture....
Query for gallery id
SELECT a.id,
a.created,
a.name,
b.clientName,
a.isFeatured,
a.views,
a.clientId
FROM gallery a
INNER JOIN client b
ON a.clientId = b.id
WHERE a.isTemp = 0
AND a.clientRef = '{$clientRef}'
AND a.finish='1'
AND a.isArchive='0'
ORDER BY created
DESC
You can consolidate the queries and eliminate the need for looping:
SELECT
a.id,
a.created,
a.name,
b.clientName,
a.isFeatured,
a.views,
a.clientId,
COALESCE(c.img_cnt, 0) AS gallery_image_count,
COALESCE(c.comment_cnt, 0) AS gallery_comment_count
FROM
gallery a
INNER JOIN
client b ON a.clientId = b.id
LEFT JOIN
(
SELECT aa.galleryId,
COUNT(DISTINCT aa.id) AS img_cnt,
COUNT(1) AS comment_cnt
FROM galleryimage aa
INNER JOIN imagecomments bb ON aa.id = bb.imgId
WHERE bb.note <> ''
GROUP BY aa.galleryId
) c ON a.id = c.galleryId
WHERE
a.isTemp = 0 AND
a.clientRef = '{$clientRef}' AND
a.finish = 1 AND
a.isArchive = 0
ORDER BY
a.created DESC
Related
I have 2 tables, borrowers and loans. I want to display on the main page the list of ALL borrowers with or without loans. If with loan, display the newest one.
I have the following sql query, basically it returns the above description except it displays the very first loan of the borrower instead of the latest one.
(Side note: I used GROUP BY to avoid duplicates. Without it the query returns duplicated borrower names if they have multiple loans. Just wanted to know if this is an efficient way of doing so.)
SELECT b.b_id,
b.isdeleted,
b.picture,
b.firstname,
b.middlename,
b.lastname,
b.address,
b.contactno,
b.birthday,
b.businessname,
b.occupation,
b.comaker,
b.comakerno,
b.remarks,
b.datecreated,
b.activeloan,
l.l_id,
l.amount,
l.payable,
l.balance,
l.mode,
l.term,
l.interestrate,
l.amortization,
l.releasedate,
l.duedate,
l.status,
l.c_id
FROM borrowers as b
LEFT JOIN loans as l ON b.b_id = l.b_id
WHERE b.isdeleted = 0
GROUP BY b.b_id
It seems the below query does exactly what i wanted.
I added the below subquery on the "ON" clause.
(SELECT MAX(l_id)
FROM jai_db.loans as l2
WHERE l2.b_id = b.b_id LIMIT 1)
SELECT b.b_id, b.isdeleted, b.picture, b.firstname, b.middlename, b.lastname, b.address, b.contactno,
b.birthday, b.businessname, b.occupation, b.comaker, b.comakerno, b.remarks, b.datecreated, b.activeloan,
l.l_id, l.amount, l.payable, l.balance, l.mode, l.term, l.interestrate, l.amortization,
l.releasedate, l.duedate, l.status, l.c_id
FROM jai_db.borrowers as b
LEFT JOIN jai_db.loans as l
ON l.l_id = (SELECT MAX(l_id)
FROM jai_db.loans as l2
WHERE l2.b_id = b.b_id LIMIT 1)
WHERE b.isdeleted = 0
I'm currently doing some query for my app and I need to get the nearest store on my current position and to do this first I need to get all the item that has the same name then get it's information and trim down that query. Now I used IN statement for this but since the items being searched are also based on a list I need to make use of another select for this here is my code so far:
select *
from product p,
store s,
branches b
where 1 = 1
and b.idproduct = p.idproduct
and p.store = s.idstore
and common_name IN(SELECT p.common_name
FROM shopping_list_content s, product p
WHERE 1 =1
AND s.iditem = p.idproduct
AND s.idlist =$listid)
Now it works as I wanted it to be but I wanted it to do the query faster than this. For now it takes more than 3 seconds for this query to run faster than this. much better if it is less than a second. Any other option I can use for this?
MySQL has difficulty optimising subqueries, when you write something like:
SELECT *
FROM T
WHERE T.ID (SELECT ID FROM T2);
It is sometimes rewritten as
SELECT *
FROM T
WHERE EXISTS
( SELECT 1
FROM T2
WHERE T.ID = T2.ID
);
The subquery is then executed once per row in T, whereas if you write:
SELECT T.*
FROM T
INNER JOIN
( SELECT DISTINCT ID
FROM T2
) T2
ON T2.ID = T.ID;
Your result set will be the same, but MySQL will first fill an in memory table with the results of the subquery and hash it on T2.ID, it then just needs to lookup against this hash table for each row in T.
Which behaviour you want really depends on how much data you are expecting from each table/subquery. If you have 1 million rows in T2, and 10 in T then there is no point in filling a temporary table with 1 million rows, only to subsequently only use it 10 times, whereas if you have a large number of rows in T and only a small amount in T2 the additional cost of materialising the subquery will be beneficial in the long run.
Another thing to point out (which has no impact on performance), the JOIN syntax you are using is the ANSI 89 syntax and was replaced by ANSI 92 explicit JOIN syntax over 20 years ago. Although directed at SQL Server, I think this article summarises the reasons to switch to the newer join syntax very well. Making your final query:
SELECT *
FROM product p,
INNER JOIN store s
ON p.store = s.idstore
INNER JOIN branches b
ON b.idproduct = p.idproduct
INNER JOIN
( SELECT DISTINCT p.common_name
FROM shopping_list_content s
INNER JOIN product p
ON s.iditem = p.idproduct
WHERE s.idlist =$listid
) s
ON s.common_name = p.common_name;
N.B. Most of the above does not apply if you are using MySQL 5.6.5 or later. In this version they introduced more Subquery Optimization that solved a lot of the above issues
This is your query fixed up to use proper join syntax:
select *
from product p join
store s
on p.store = s.idstore join
branches b
on b.idproduct = p.idproduct
where p.common_name IN (SELECT p.common_name
FROM shopping_list_content slc join
product p
ON slc.iditem = p.idproduct AND
slc.idlist = $listid
);
Assuming that the same common_name does not appear on multiple products and that shopping_list_content has no duplicate rows, you can replace this with a simple join:
select *
from product p join
store s
on p.store = s.idstore join
branches b
on b.idproduct = p.idproduct join
shopping_list_content slc
on slc.iditem = p.idproduct and
slc.idlist = $listid;
However, those assumptions may not be true. In that case, changing the subquery to use exists may help performance:
select *
from product p join
store s
on p.store = s.idstore join
branches b
on b.idproduct = p.idproduct
where exists (SELECT 1
FROM shopping_list_content slc join
product p2
on slc.iditem = p2.idproduct AND
slc.idlist = $listid
WHERE p.common_name = p2.common_name
);
For this latter query, an index on product(common_name, idproduct) along with shopping_list_content(iditem, idlist) should help.
I currently have this left join as part of a query:
LEFT JOIN movies t3 ON t1.movie_id = t3.movie_id AND t3.popularity = 0
The trouble is that if there are several movies with the same name and same popularity (don't ask, it just is that way :-) ) then duplicate results are returned.
All that to say, I would like to limit the result of the left join to one.
I tried this:
LEFT JOIN
(SELECT t3.movie_name FROM movies t3 WHERE t3.popularity = 0 LIMIT 1)
ON t1.movie_id = t3.movie_id AND t3.popularity = 0
The second query dies with the error:
Every derived table must have its own alias
I know what I'm asking is slightly vague since I'm not providing the full query, but is what I'm asking generally possible?
The error is clear -- you just need to create an alias for the subquery following its closing ) and use it in your ON clause since every table, derived or real, must have its own identifier. Then, you'll need to include movie_id in the subquery's select list to be able to join on it. Since the subquery already includes WHERE popularity = 0, you don't need to include it in the join's ON clause.
LEFT JOIN (
SELECT
movie_id,
movie_name
FROM movies
WHERE popularity = 0
ORDER BY movie_name
LIMIT 1
) the_alias ON t1.movie_id = the_alias.movie_id
If you are using one of these columns in the outer SELECT, reference it via the_alias.movie_name for example.
Update after understanding the requirement better:
To get one per group to join against, you can use an aggregate MAX() or MIN() on the movie_id and group it in the subquery. No subquery LIMIT is then necessary -- you'll receive the first movie_id per name withMIN() or the last with MAX().
LEFT JOIN (
SELECT
movie_name,
MIN(movie_id) AS movie_id
FROM movies
WHERE popularity = 0
GROUP BY movie_name
) the_alias ON t1.movie_id = the_alias.movie_id
LEFT JOIN movies as m ON m.id = (
SELECT id FROM movies mm WHERE mm.movie_id = t1.movie_id
ORDER BY mm.id DESC
LIMIT 1
)
you could try to add GROUP BY t3.movie_id to the first query
Try this:
LEFT JOIN
(
SELECT t3.movie_name, t3.popularity
FROM movies t3 WHERE t3.popularity = 0 LIMIT 1
) XX
ON t1.movie_id = XX.movie_id AND XX.popularity = 0
On MySQL 5.7+ use ANY_VALUE & GROUP_BY:
SELECT t1.id,t1.movie_name, ANY_VALUE(t3.popularity) popularity
FROM t1
LEFT JOIN t3 ON (t3.movie_id=t1.movie_id AND t3.popularity=0)
GROUP BY t1.id
more info
LEFT JOIN only first row
https://dev.mysql.com/doc/refman/5.7/en/group-by-handling.html
Easy solution to left join the 1 most/least recent row is using select over ON phrase
SELECT A.ID, A.Name, B.Content
FROM A
LEFT JOIN B
ON A.id = (SELECT MAX(id) FROM B WHERE id = A.id)
Where A.id is the auto-incremental primary key.
LEFT JOIN (
SELECT id,movie_name FROM movies GROUP BY id
) as m ON (
m.id = x.id
)
// Mysql
SELECT SUM(db.item_sales_nsv) as total FROM app_product_hqsales_otc as db
LEFT JOIN app_item_target_otc as it ON
db.id = (SELECT MAX(id) FROM app_item_target_otc as ot WHERE id = db.id)
and db.head_quarter = it.hqcode
AND db.aaina_item_code = it.aaina_item_code AND db.month = it.month
AND db.year = it.year
WHERE db.head_quarter = 'WIN001' AND db.month = '5' AND db.year = '2022' AND db.status = '1'
I use a query like :
$querym = mysql_query("SELECT * FROM allmembers a LEFT JOIN favorites f ON (f.memberid=a.memberid) order by f.date desc LIMIT 10");
while($row = mysql_fetch_array($querym,MYSQL_ASSOC)) {
$dataArray[$row['memberid']][$row['favoriteid']]=$row;
}
My purpose is getting 10 members with their last 5 favorites in an array. but as you can guess this query getting 10 row included with favorites. That means if a member have 15 favorites it only gets one member with 10 favorites instead of 10 members with his favorites.
I couldn't find an easy way to limit getting favorites for each member in that query. How can I limit?
Thanks in advance
This is not MYSQL FETCH ARRAY LIMITATION, this is your QUERY limitation.
Try something of your own on these lines :
SELECT memberid,group_concat(favorites_field) FROM allmembers a LEFT JOIN favorites f ON (f.memberid=a.memberid) group by a.memberid order by f.date desc LIMIT 10
This query may give you a better approach
SELECT * FROM favorites f where f.memberid in (SELECT * FROM allmembers limit 10) ORDER BY f.date desc
although you should limit the 5 favorites later, in php or whatever. This could work if you don't expect to have 10.000 favorites for one member (basically because you will be getting 10.000+ ROWS just to display, at maximum, 10*5 = 50 rows.)
Alternatively, you can do a query for each member, limiting it to 5 results... (this would mean doing 11 queries in total...)
The following query will do what you want:
SELECT
a.id as member_id,
a.name,
f1.id as favorite_id,
f1.link
FROM
allmembers a
JOIN favorites f1 ON f1.member = a.id
LEFT JOIN favorites f2 ON f2.member = f1.member AND f2.date < f1.date
GROUP BY
a.id, a.name, f1.id, f1.link
HAVING
COUNT(f2.id) < 10
ORDER BY
a.name, f1.id
It assumes the following database schema:
allmembers:
id INT
name VARCHAR
favorites:
id INT
member INT
link VARCHAR
Obviously you need to update the query according to your own database schema.
You have many ways to do that, and they can all be performant up to a limit, what i'd do in this specific case is this:
Get the favorite id's in a multiple left join as a seperate field, you can then easily for(...) the row array for each field and retrieve the id. The Query would LOOK LIKE this. Adapt it to your own use:
SELECT
a.*,
f1.favoriteid as f1id,
f2.favoriteid as f2id,
f3.favoriteid as f3id,
f4.favoriteid as f4id,
f5.favoriteid as f5id
FROM allmembers a
LEFT JOIN favorites f1 ON (f.memberid=a.memberid)
LEFT JOIN favorites f2 ON (f.memberid=a.memberid) AND f1.favoriteid <> f2.favoriteid
LEFT JOIN favorites f3 ON (f.memberid=a.memberid) AND f1.favoriteid <> f3.favoriteid AND f2.favoriteid <> f3.favoriteid
LEFT JOIN favorites f4 ON (f.memberid=a.memberid) AND f1.favoriteid <> f4.favoriteid AND f2.favoriteid <> f4.favoriteid AND f3.favoriteid <> f4.favoriteid
LEFT JOIN favorites f5 ON (f.memberid=a.memberid) AND f1.favoriteid <> f5.favoriteid AND f2.favoriteid <> f5.favoriteid AND f3.favoriteid <> f5.favoriteid AND f4.favoriteid <> f5.favoriteid
order by f.date desc LIMIT 10
Using this method you can also read any information from the favorites table or even left join on another table X number of times to get more information regarding that favorite. As long as you setup correct indexes, this method is extremely fast even with thousands of members and nearly millions of favorites.
You can also apply this strategy to many other scenarios. For example, we work with WordPress here at work and lots of information for users are kept as meta fields, so selecting one big table is impossible unless you perform this method.
Good luck
I'm not quite sure if this is the right approach, this is my situation:
I'm currently trying to select 15 galleries and then left join it with the user table through the id but I also want to select one random picture from each gallery however from what I know you can't limit the left join (picture) to only pick up one random picture without doing a subquery.
Here is what I got so far but its not working as it should:
SELECT galleries.id, galleries.name, users.username, pictures.url
FROM galleries
LEFT JOIN users ON users.id = galleries.user_id
LEFT JOIN pictures ON (
SELECT pictures.url
FROM pictures
WHERE pictures.gallery_id = galleries.id
ORDER BY RAND()
LIMIT 1)
WHERE active = 1
ORDER BY RAND()
LIMIT 15
I also tried to do this with Active Record but I got stuck after doing two left joins, is it possible to do get a subquery in here:
$this->db->select('galleries.id, galleries.name, users.id as user_id, users.username');
$this->db->from('galleries');
$this->db->join('users', 'users.id = galleries.user_id','left');
$this->db->join('pictures','pictures.gallery_id = galleries.id AND','left');
$this->db->where('active',1);
I hope its not to messy but I'm really starting to get confusing by all the sql queries..
Edit:
Active Record with CodeIgniter
You could fetch a random picture in a subquery:
select
g.name, u.username,
(select url from pictures p where p.gallery_id = g.gallery_id
order by rand() limit 1) as url
from galleries g
left join users u on g.user_id = u.id
where g.active = 1
Based on your comment, you could select a picture for each gallery in a subquery. This is assuming the picture table has an ID column.
select
g.name, u.username, p.url, p.name
from (
select id, user_id, name,
(select id from pictures p
where p.gallery_id = g.gallery_id
order by rand() limit 1) as samplepictureid
from galleries
where g.active = 1
) g
left join users u on g.user_id = u.id
left join pictures p on p.id = g.samplepictureid
SELECT
g.id,
g.name,
u.username,
p.url
FROM
galleries g
INNER JOIN (SELECT DISTINCT
gallery_id,
(SELECT url FROM pictures ss WHERE ss.gallery_id = s.gallery_id
ORDER BY RAND() LIMIT 1) AS url
FROM
pictures s) p ON
g.id = p.gallery_id
LEFT OUTER JOIN users u ON
g.user_id = u.id
WHERE
g.active = 1
This query will go out and select a gallery, then it will find any gallery with a picture (if you want to return galleries without a picture, change INNER JOIN to LEFT OUTER JOIN, and you'll be fine). After that, it joins it up with users. Now, of course, this puppy is going to return every frigging gallery for however many users you have (hoorah!). You may want to limit the user in the WHERE clause (e.g.-WHERE u.id = 123). Otherwise, you're going to get more results than you'd expect. That, or do an INNER JOIN on it.