I've got a MySQL query that runs very, very slowly and pegs the server usage at 100% as soon as it's executed....sometimes.
Here's the query:
SELECT DISTINCT r1.recordID,
(SELECT MAX(r2.date)
FROM reminders as r2
WHERE r2.owner = '$owner'
AND r2.recordID = r1.recordID
AND r2.status = 'Active'
AND r2.followUp != 'true'
ORDER BY r2.date DESC LIMIT 1) as maxDate
FROM reminders as r1
WHERE r1.owner = '$owner'
AND (SELECT MAX(r2.date)
FROM reminders as r2
WHERE r2.recordID = r1.recordID
AND r2.status = 'Active'
AND r2.followUp != 'true'
ORDER BY r2.date DESC LIMIT 1) <= '$date'
AND (SELECT do_not_call
FROM marketingDatabase
WHERE id = r1.recordID) != 'true'
AND r1.status = 'Active'
ORDER BY maxDate DESC
I'm not sure if it's a poorly written query (might be...I'm new at this) or something else. Sometimes it works fine and results are returned almost instantly but other times it takes a long time (15+ minutes) and 100% server resources to return the results.
Any idea why this could be happening? Anything I can do to the query to prevent this?
Thanks in advance!
[EDIT]
Here's the EXPLAIN.
Array
(
[0] => Array
(
[id] => 1
[select_type] => PRIMARY
[table] => r1
[type] => ALL
[possible_keys] =>
[key] =>
[key_len] =>
[ref] =>
[rows] => 2073
[Extra] => Using where; Using temporary; Using filesort
)
[1] => Array
(
[id] => 4
[select_type] => DEPENDENT SUBQUERY
[table] => marketingDatabase
[type] => eq_ref
[possible_keys] => PRIMARY
[key] => PRIMARY
[key_len] => 4
[ref] => teleforce.r1.recordID
[rows] => 1
[Extra] =>
)
[2] => Array
(
[id] => 3
[select_type] => DEPENDENT SUBQUERY
[table] => r2
[type] => ALL
[possible_keys] =>
[key] =>
[key_len] =>
[ref] =>
[rows] => 2073
[Extra] => Using where
)
[3] => Array
(
[id] => 2
[select_type] => DEPENDENT SUBQUERY
[table] => r2
[type] => ALL
[possible_keys] =>
[key] =>
[key_len] =>
[ref] =>
[rows] => 2073
[Extra] => Using where
)
)
Several things could make this query slow.
One: In general, the first thing to check is indexes. You have three embedded queries that have to be executed for each record in r1. If any of these are not able to effectively use an index and have to process a large number of records, this query will be very slow. Review your indexes, and use "explain" to see what's being used.
Two: Embedded queries tend to be slower than joins. See if you can't transform some of your embedded queries to joins.
Three: In this particular query, you're joining "remainders" back on itself, as far as I can figure out, just to find max(date). Why not just use a GROUP BY? My step 1 to improving this query would be:
select r1.recordID, max(r1.date) as maxdate
from reminders as r1
where r1.owner=$owner and r1.status='Active' and r1.followUp!='true'
and (SELECT do_not_call FROM marketingDatabase WHERE id = r1.recordID) != 'true'
group by r1.recordID
having max(r1.date)<=$date
order by maxdate desc
I don't have your database to test this, but I think it would give the same results.
Four: I'd turn the other embedded query into a join. Like:
select r1.recordID, max(r1.date) as maxdate
from reminders as r1
join marketingDatabase as m on m.id=r1.recordID
where r1.owner=$owner and r1.status='Active' and r1.followUp!='true'
and m.do_not_call != 'true'
group by r1.recordID, r1.owner
having max(r1.date)<=$date
order by maxdate desc
(I'm not sure what a recordID identifies. It appears from your query that you can have multiple records in reminders with the same recordid.)
Five: You'll probably get best performance if you have indexes on reminders(owner, date) and marketingDatabase(id).
Six: Just by the way, if "do_not_call" and followUp are true/false, they should be booleans and not varchars. You're just wasting disk space and execution time processing "true" instead of a boolean TRUE. And you create the problem of mis-spellings, like "ture" or "True". At the absolute worst, make them a char(1), T or F.
There are times when you need embedded queries, but they should be a last resort.
Try to create an index on reminders.owner with the command
CREATE INDEX someNameYouChoose ON reminders(owner);
Related
I have query like this,
SELECT * FROM users ORDER BY score
So, the result is like this.
Array
(
[0] => stdClass Object
(
[userid] => 3
[user] => John Doe
[score] => 50
)
[1] => stdClass Object
(
[userid] => 1
[user] => Mae Smith
[score] => 38
)
[2] => stdClass Object
(
[userid] => 2
[user] => Mark Sam
[score] => 26
)
)
But, I want to add a rank using find_in_set query. So the result might be like this. So that the user can view their ranks when they login to their account.
Array
(
[0] => stdClass Object
(
[userid] => 3
[user] => John Doe
[score] => 50
[rank] => 1
)
[1] => stdClass Object
(
[userid] => 1
[user] => Mae Smith
[score] => 38
[rank] => 2
)
[2] => stdClass Object
(
[userid] => 2
[user] => Mark Sam
[score] => 26
[rank] => 3
)
)
I tried this one.
$listOfUser = array();
foreach($users as $user) {
$listOfUser[] = $user->userid;
}
And used another query
$userid = 2 // => id of loggedin user
SELECT *, find_in_set($userid, $listOfUser) as rank FROM users where userid=$userid ORDER BY score
So, I got this result
Array
(
[1] => stdClass Object
(
[userid] => 2
[user] => Mark Sam
[score] => 26
[rank] => 3
)
)
Which is somehow correct. But, is there another way of querying that result using only one SQL query and without using foreach loop?
Something like this.
$userid = 2 // => id of loggedin user
SELECT *, find_in_set($userid, (SELECT * FROM users ORDER BY score)) as rank FROM users where userid=$userid ORDER BY score
But I got this error Subquery returns more than 1 row
If You don't insist on using find_in_set, you can get result with simple join. You ask for list of users (p) and for each user you ask, how many users have better score than him or her (c):
SELECT p.userid, COUNT(c.userid) AS rank
FROM users AS p
LEFT JOIN users AS c ON c.score > p.score
GROUP BY p.userid
This works even if you add other conditions, like WHERE p.userid = 123.
If more users have the same score, the ranks would look like 0,1,2,2,2,5,6.
In your query, you can add counter, like this:
set #n:=0;
SELECT #i := #i + 1 AS rank, * FROM users ORDER BY score
The rank here is relative to the score distribution across all users. I believe you should try something originally proposed in this answer:
SELECT users.*,
#rownum := #rownum + 1 as rank
FROM users
CROSS JOIN (select #rownum := 0) r
ORDER BY score DESC
What it does is basically order all users by score, and assign each of them an incremental value "rank". So the top scorer would have a rank of 1, the second scorer would have a rank of 2 etc.
Keep in mind that this solution is not "fair" - each user will have a different rank, even if all users have the same score. If you try to rank users as they do in sports (if two top competitors have the same score, they both take 1st place, and the next best competitor takes 3rd place, not second), you should think of a different solution.
I am using corequery of mysql in cakephp. I want the records in descending order. This is my table structure
enter code here
$coreQueryUser = $this->Message->query(
"select * from messages where messages.list_id = 3
group By (if(sender_id > reciever_id, sender_id, reciever_id)),
(if(sender_id > reciever_id, reciever_id, sender_id))
order by id desc
"
);
I want last message that belongs to (sender_id and reciver_id and viceversa) that belongs to list id 3
when i run this query i get the following output
<pre>Array
(
[0] => Array
(
[messages] => Array
(
[id] => 1
[sender_id] => 21
[reciever_id] => 10
[list_id] => 3
[message] => hello
[add_date] => 2016-09-25 00:00:00
[is_check] => 0
)
)
[1] => Array
(
[messages] => Array
(
[id] => 3
[sender_id] => 22
[reciever_id] => 10
[list_id] => 3
[message] => hello s
[add_date] => 2016-09-25 16:39:41
[is_check] => 0
)
)
)
but i wnat result like that:
Array
(
[0] => Array
(
[messages] => Array
(
[id] => 2
[sender_id] => 10
[reciever_id] => 21
[list_id] => 3
[message] => hello sir
[add_date] => 2016-09-25 00:00:00
[is_check] => 0
)
)
[1] => Array
(
[messages] => Array
(
[id] => 6
[sender_id] => 22
[reciever_id] => 10
[list_id] => 3
[message] => new
[add_date] => 2016-09-25 16:39:41
[is_check] => 0
)
)
)
Can anyone help me :(
The problem is that your query is against the sql standard because you have several fields in the select list that are neither in the group by list, nor are subject of an aggregate function, such as sum(). MySQL unfortunately allows such invalid queries to run under certain sql mode settings (the default settings of the most recent versions of MySQL would prevent such queries from running).
As MySQL documentation on group by clause says (bolding is mine):
If ONLY_FULL_GROUP_BY is disabled, a MySQL extension to the standard
SQL use of GROUP BY permits the select list, HAVING condition, or
ORDER BY list to refer to nonaggregated columns even if the columns
are not functionally dependent on GROUP BY columns. This causes MySQL
to accept the preceding query. In this case, the server is free to
choose any value from each group, so unless they are the same, the
values chosen are indeterminate, which is probably not what you want.
Furthermore, the selection of values from each group cannot be
influenced by adding an ORDER BY clause. Result set sorting occurs
after values have been chosen, and ORDER BY does not affect which
value within each group the server chooses.
You apparently want the latest record (with max(id) for each group. The proper way is to have a subquery that returns the max(id) per group and in the outer query join back to your main table using the ids to get the value of the other fields:
select m.*
from
messages m
inner join (
select max(id) as maxid
from messages
where messages.list_id = 3
group By (if(sender_id > reciever_id, sender_id, reciever_id)),
(if(sender_id > reciever_id, reciever_id, sender_id))
) t1 on m.id=t1.maxid
This code working:
SELECT * FROM generate_invoice
WHERE id IN
(
SELECT max(id) as id
FROM generate_invoice
GROUP by pay_id
ORDER by id DESC
)
How to group by DESC order
Try
SELECT * FROM ( SELECT * FROM generate_invoice ORDER BY id DESC ) AS g GROUP BY g.pay_id
OR
Use this code
SELECT m1.*,m2.* FROM generate_invoice m1 LEFT JOIN generate_invoice m2 ON (m1.pay_id = m2.pay_id AND m1.id < m2.id ) order by m1.id desc
on my ratings table for my software i have 4 fields.
id autoincrement
rvid vendor id
ratedate date of rating
rating the actual numeric rating
I have done alot with it over the last few months but this time im stumped and i cant get a clear picture in my head of the best way to do this. What i am trying to do is find out if the vendor has had 3 low 'consecutive' ratings. If their last three ratings have been < 3 then i want to flag them.
I have been playing with this for a few hours now so i thought i would ask (not for the answer) but for some path direction just to push me forward, im stuck in thought going in circles here.
I have tried GROUP BY and several ORDER BY but those attempts did not go well and so i am wondering if this is not a mysql answer but a php answer. In other words maybe i just need to take what i have so far and just move to the php side of things via usort and the like and do it that way.
Here is what i have so far i did select id as well at first thinking that was the best way to get the last consective but then i had a small breakthrough that if they have had 3 in a row the id does not matter, so i took it out of the query.
$sql = "SELECT `rvid`, `rating` FROM `vendor_ratings_archive` WHERE `rating` <= '3' ORDER BY `rvid` DESC";
which give me this
Array
(
[0] => Array
(
[rvid] => 7
[rating] => 2
)
[1] => Array
(
[rvid] => 5
[rating] => 1
)
[2] => Array
(
[rvid] => 5
[rating] => 0
)
[3] => Array
(
[rvid] => 5
[rating] => 3
)
)
this is just just samples i tossed in the fields, and there are only 4 rows here where as in live it will be tons of rows. But basically this tells me that these are the vendors that have low ratings in the table. And that is where i get stumpted. I can only do one sort in the query so that is why i am thinking that i need to take this and move to the php side to finish it off.
I think i need to sort the elements by rvid with php first i think, and then see if three elements in a row are the same vender (rvid).
Hope that makes sense. My brain hurts lol...
update - here is all of the table data using *
Array
(
[0] => Array
(
[id] => 7
[rvid] => 7
[ratedate] => 2016-05-01
[rating] => 2
)
[1] => Array
(
[id] => 8
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 1
)
[2] => Array
(
[id] => 6
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 0
)
[3] => Array
(
[id] => 5
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 3
)
)
Here's one way you can begin approaching this - completely in SQL:
Get the last rating for the vendor. ORDER BY date DESC, limit 1.
Get the second to last rating for the vendor. ORDER BY date DESC, limit 1, OFFSET 1.
Then write a query that does a LEFT join of the first two tables. You will have a dataset that has three columns:
vendor id
last rating
second to last rating
Then you can write an expression that says "if column1 is <3 and column2 < 3, then this new column is true"
You should be able to extend this to three columns relatively easily.
Here is what a came up with to solve this riddle. I think explaining it on here helped as well as Alex also helped as he keyed my brain on using the date. I first started looking at using if statment inside of the query and actually that got my brain out of the box and then it hit me what to do.
It is not perfect and certainly could use some trimming to reduce the code, but i understand it and it seems to work, so that is par for me on this course.
the query...
$sql = "SELECT `rvid`, `ratedate`,`rating` FROM `vendor_ratings_archive` WHERE `rating` <= '3' ORDER BY `ratedate`, `rvid` DESC";
which gives me this
Array
(
[0] => Array
(
[rvid] => 7
[ratedate] => 2016-05-01
[rating] => 2
)
[1] => Array
(
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 1
)
[2] => Array
(
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 0
)
[3] => Array
(
[rvid] => 5
[ratedate] => 2016-05-01
[rating] => 3
)
)
notice how vendor (rvid) 5 is grouped together which is an added plus.
next a simple foreach to load a new array
foreach($results as $yield)
{
$rvidarray[] = $yield['rvid'];
}//close foreach
which gives me this
Array
(
[0] => 7
[1] => 5
[2] => 5
[3] => 5
)
then we count the array values to group dups
$rvidcounter = array_count_values($rvidarray);
which results in this
Array(
[7] => 1
[5] => 3
)
so now vender 7 as 1 low score and vendor 5 has 3 low scores and since they were already sorted by date i know that its consecutive. Well it sounds good anyway lol ")
then we create our final array with another foreach
foreach($rvidcounter as $key => $value)
{
//anything 3 or over is your watchlist
if($value > 2)
{
$watchlist[] = $key; //rvid number stored
}
}//close foreach
which gives me this
Array
(
[0] => 5
)
this was all done in a service function. So the final deal is everyone in this array has over 3 consecutive low ratings and then i just use a returned array back in my normal php process file and grab the name of each vender by id and pass that to the html and print out the list.
done...
please feel free to improve on this if you like. I may or may not use it because the above code makes sense to me. Something more complicated may not make sense to me 6 mos from now lol But it would be interesting to see what someone comes up with to shorten the process a bit.
Thanks so much and Happy Coding !!!!!
Dave :)
You could do it in SQL like that:
SET #rvid = -1;
SELECT DISTINCT rvid FROM
(
SELECT
rvid,
#neg := rating<3, /* 0 or 1 for addition in next line */
#count := IF(#rvid <> rvid , #neg, #count+#neg) AS `count`, /* set or add */
#rvid := rvid /* remember last row */
FROM
testdb.venrate
ORDER BY
rvid, datetime desc
) subq
WHERE count>=3
;
You set a variable to a non existing id. In each chronologically sorted row you check if rating is too low, that results in 1 (too low) or 0 (ok). If rvid is not equal to the last rvid, it means a new vender section is beginning. On begin of section set the value 0 or 1, else add this value. Finally store the current row's rvid for comparison in next row process.
The code above is looking for 3 consecutive low ratings (low means a value less than 3) over all the time.
A small modification checks if all the latest 3 ratings has been equal to or less than 3:
SET #rvid = -1;
SELECT DISTINCT
rvid
FROM
(
SELECT
rvid,
#high_found := rating>3 OR (#rvid = rvid AND #high_found) unflag,
#count := IF(#rvid <> rvid , 1, #count+1) AS `count`,
#rvid := rvid /* remember last row */
FROM
testdb.venrate
ORDER BY
rvid, datetime desc
) subq
WHERE count=3 AND NOT unflag
;
I’ve seen the following question on StackOverflow, Intelligent MySQL GROUP BY for Activity Streams posted by Christian Owens 12/12/12.
So I decided to try out the same approach, make two tables similar to those of his. And then I pretty much copied his query which I do understand.
This is what I get out from my sandbox:
Array
(
[0] => Array
(
[id] => 0
[user_id] => 1
[action] => published_post
[object_id] => 776286559146635
[object_type] => post
[stream_date] => 2015-11-24 12:28:09
[rows_in_group] => 1
[in_collection] => 0
)
)
I am curious, since looking at the results in Owens question, I am not able to fully get something, and does he perform additional queries to grab the actual metadata? And if yes, does this mean that one can do it from that single query or does one need to run different optimized sub-queries and then loop through the arrays of data to render the stream itself.
Thanks a lot in advanced.
Array
(
[0] => Array
(
[id] => 0
[user_id] => 1
[fullname] => David Anderson
[action] => hearted
[object_id] => array (
[id] => 3438983
[title] => Grand Theft Auto
[Category] => Games
)
[object_type] => product
[stream_date] => 2015-11-24 12:28:09
[rows_in_group] => 1
[in_collection] => 1
)
)
In "pseudo" code you need something like this
$result = $pdo->query('
SELECT stream.*,
object.*,
COUNT(stream.id) AS rows_in_group,
GROUP_CONCAT(stream.id) AS in_collection
FROM stream
INNER JOIN follows ON stream.user_id = follows.following_user
LEFT JOIN object ON stream.object_id = object.id
WHERE follows.user_id = '0'
GROUP BY stream.user_id,
stream.verb,
stream.object_id,
stream.type,
date(stream.stream_date)
ORDER BY stream.stream_date DESC
');
then parse the result and convert it in php
$data = array(); // this will store the end result
while($row = $result->fetch(PDO::FETCH_ASSOC)) {
// here for each row you get the keys and put it in a sub-array
// first copy the selected `object` data into a sub array
$row['object_data']['id'] = $row['object.id'];
$row['object_data']['title'] = $row['object.title'];
// remove the flat selected keys
unset($row['object.id']);
unset($row['object.title']);
...
$data[] = $row; // move to the desired array
}
you should get
Array
(
[0] => Array
(
[id] => 0
[user_id] => 1
[fullname] => David Anderson
[verb] => hearted
[object_data] => array (
[id] => 3438983
[title] => Grand Theft Auto
[Category] => Games
)
[type] => product
[stream_date] => 2015-11-24 12:28:09
[rows_in_group] => 1
[in_collection] => 1
)
)
It seems that you want a query where you can return the data you're actually able to get plus the user fullname and the data related to the object_id.
I think that the best effort would be to include some subqueries in your query to extract these data:
Fullname: something like (SELECT fullname FROM users WHERE id = stream.user_id) AS fullname... or some modified version using the stream.user_id, as we can't identify in your schema where this fullname comes from;
Object Data: something like (SELECT CONCAT_WS(';', id, title, category_name) FROM objects WHERE id = stream.object_id) AS object_data. Just as the fullname, we can't identify in your schema where these object data comes from, but I'm assuming it's an objects table.
One object may have just one title and may have just one category. In this case, the Object Data subquery works great. I don't think an object can have more than one title, but it's possible to have more than one category. In this case, you should GROUP_CONCAT the category names and take one of the two paths:
Replace the category_name in the CONCAT_WS for the GROUP_CONCAT of all categories names;
Select a new column categories (just a name suggestion) with the subquery which GROUP_CONCAT all categories names;
If your tables were like te first two points of my answer, a query like this may select the data, just needing a proper parse (split) in PHP:
SELECT
MAX(stream.id) as id,
stream.user_id,
(select fullname from users where id = stream.user_id) as fullname,
stream.verb,
stream.object_id,
(select concat_ws(';', id, title, category_name) from objects where id = stream.object_id) as object_data,
stream.type,
date(stream.stream_date) as stream_date,
COUNT(stream.id) AS rows_in_group,
GROUP_CONCAT(stream.id) AS in_collection
FROM stream
INNER JOIN follows ON 1=1
AND stream.user_id = follows.following_user
WHERE 1=1
AND follows.user_id = '0'
GROUP BY
stream.user_id,
stream.verb,
stream.object_id,
stream.type,
date(stream.stream_date)
ORDER BY stream.stream_date DESC;
In ANSI SQL you can't reference columns not listed in your GROUP BY, unless they're in aggregate functions. So, I included the id as an aggregation.
I'm doing this query from any mysql client, and it's working:
SELECT
q.calldate,
q.dst,
#ts_dnd_activate := IF(q.dst='*78',q.calldate,#ts_dnd_activate) AS dndact,
#ts_dnd_deactivate := IF(q.dst='*79',q.calldate,NULL) AS dnddeact,
TIMESTAMPDIFF(SECOND,#ts_dnd_activate,#ts_dnd_deactivate) AS tdiff
FROM
(SELECT calldate,dst
FROM cdr
WHERE calldate>='2012-01-09 11:12:08' AND src=1004 AND dst IN ('*78','*79')
ORDER BY calldate DESC) q
It gets two fields from a table and, based on the value of the dst field, uses mysql variables to calculate the difference between the calldates fields, which is a timestamp.
If I run this query in PHP code:
[...]
$q = "SELECT
q.calldate,
q.dst,
#ts_dnd_activate := IF(q.dst='*78',q.calldate,#ts_dnd_activate) AS dndact,
#ts_dnd_deactivate := IF(q.dst='*79',q.calldate,NULL) AS dnddeact,
TIMESTAMPDIFF(SECOND,#ts_dnd_activate,#ts_dnd_deactivate) AS tdiff
FROM
(SELECT calldate,dst
FROM cdr
WHERE calldate>='2012-01-09 11:12:08' AND src=1004 AND dst IN ('*78','*79')
ORDER BY calldate DESC) q";
$rs = mysql_query($q,$database);
[...]
I'll never get the tdiff field, because I'm getting rows where the mysql variables are always NULL, like this:
Array
(
[calldate] => 2012-01-09 12:20:10
[dst] => *78
[dndact] => 2012-01-09 12:20:10
[dnddeact] =>
[tdiff] =>
)
Array
(
[calldate] => 2012-01-09 12:22:05
[dst] => *79
[dndact] =>
[dnddeact] => 2012-01-09 12:22:05
[tdiff] =>
)
Array
(
[calldate] => 2012-01-09 12:25:15
[dst] => *78
[dndact] => 2012-01-09 12:25:15
[dnddeact] =>
[tdiff] =>
)
Array
(
[calldate] => 2012-01-09 12:29:21
[dst] => *79
[dndact] =>
[dnddeact] => 2012-01-09 12:29:21
[tdiff] =>
)
Please note that it is a SINGLE query, but it seems that it's impossible to use mysql variables with the php mysql base extension because I don't know why but it seems to set to null every row.
What I'm doing wrong? Is it a new problem?
P.S. please don't tell me I've to use PDO or mysqli; its an old project that I can't rewrite.