I have a big data problem with MySQL.
I have:
a users table with 59033 rows, and
a user_notes table with 8753 rows.
But when I search which users have user note in some dates.
My query like this :
SELECT u.*, rep.name as rep_name FROM users as u
LEFT JOIN users as rep on rep.id = u.add_user
LEFT JOIN authorization on authorization.id = u.authorization
LEFT JOIN user_situation_list on user_situation_list.user_situation_id = u.user_situation
WHERE
EXISTS(
select * from user_notes
where user_notes.note_user_id = u.id AND user_notes.create_date
BETWEEN "2017-10-20" AND "2017-10-22"
)
ORDER BY u.lp_modify_date DESC, u.id DESC
Turn it around -- find the ids first; deal with the joins later.
SELECT u.*,
( SELECT rep.name
FROM users AS rep
WHERE rep.id = u.add_user ) AS rep_name
FROM (
SELECT DISTINCT note_user_id
FROM user_notes
WHERE create_date >= "2017-10-20"
AND create_date < "2017-10-20" + INTERVAL 3 DAY
) AS un
JOIN users AS u ON u.id = un.note_user_id
ORDER BY lp_modify_date DESC, id DESC
Notes
No GROUP BY needed;
2 tables seem to be unused; I removed them;
I changed the date range;
User notes needs INDEX(create_date, note_user_id);
Notice how I turned a LEFT JOIN into a subquery in the SELECT list.
If there can be multiple rep_names, then the original query is "wrong" in that the GROUP BY will pick a random name. My Answer can be 'fixed' by changing rep.name to one of these:
MAX(rep.name) -- deliver only one; arbitrarily the max
GROUP_CONCAT(rep.name) -- deliver a commalist of names
Rewriting your query to use a JOIN rather than an EXISTS check in the where should speed it up. If you then group the results by the user.id it should give you the same result:
SELECT u.*, rep.name as rep_name FROM users as u
LEFT JOIN users as rep on rep.id = u.add_user
LEFT JOIN authorization on authorization.id = u.authorization
LEFT JOIN user_situation_list on user_situation_list.user_situation_id = u.user_situation
JOIN user_notes AS un
ON un.note_user_id
AND un.create_date BETWEEN "2017-10-20" AND "2017-10-22"
GROUP BY u.id
ORDER BY u.lp_modify_date DESC, u.id DESC
I had a SELECT query with a LEFT JOIN working as desired. I then added one more table via a smilar LEFT JOIN and now I am getting a wierd result. Basically, for a group_concat where I was getting one item for every record, I am getting eight records. I don't see why this is happening because the join to the new table is analagous to several other joins that do not have this problem (that I have omitted from the example for clarity).
Here is the query that is fine:
$sql = "SELECT t.*,
group_concat(tf.todoid) as `tftodoid`,
group_concat(tf.id) as `tfid`,
group_concat(tf.filedescript) as `tffiledescript`,
group_concat(tf.filename) as `tffilename`,
group_concat(tf.founderid) as `tffounderid`,
group_concat(tf.ext) as `tfext`,
group_concat(tf.lasttouched) as `tilt`
FROM titles `t`
LEFT JOIN titlefiles `tf`
ON (tf.todoid = t.id AND tf.founderid = '$userid')
WHERE t.userid='$userid'
GROUP BY t.id";
And here is the query with the extra join that is now spilling out the multiple copies of the items:
$sql = "SELECT t.*,
group_concat(tf.todoid) as `tftodoid`,
group_concat(tf.id) as `tfid`,
group_concat(tf.filedescript) as `tffiledescript`,
group_concat(tf.filename) as `tffilename`,
group_concat(tf.founderid) as `tffounderid`,
group_concat(tf.ext) as `tfext`,
group_concat(tf.lasttouched) as `tilt`,
group_concat(s.id) as `stepid`,
group_concat(s.step) as `steps`
FROM titles `t`
LEFT JOIN titlefiles `tf`
ON (tf.titleid = t.id AND tf.founderid = '$userid')
LEFT JOIN steps `s`
ON s.titleid = t.id
WHERE t.userid='$userid'
GROUP BY t.id";
Here is an example of output in JSON showing the difference:
First query:
"tfid":"56,57,58,59,60,61,62,63,64,65,66,67,68,75,76,81"
Second query:
"tfid":"56,57,58,59,60,61,62,63,64,65,66,67,68,75,76,81,56,57,58,59,60,61,62,63,64,65,66,67,68,75,76,81,56,57,58,59,60,61,62,63,64,65,66,67,68,75,76,81,56,57,58,59,60,61,62,63,64,65,66,67,68,75,76,81,56,57,58,59,60,61,62,63,64,65,66,67,68,75,76,81,56,57,58,59,60,61,62,63,64,65,66,67,68,75,76,81,56,57,58,59,60,61,62,63,64,65,66,67,68,75,76,81,56,57,58,59,60,61,62,63,64,65,66,67,68,75,76,81,56,57,58,59,60,61,62,63,64,65,66,67,68,75,76,81",
I suspect the problem has something to do with the JOIN or with the Group By statements but I can't see how to fix.
How can I ensure that I get only one fileid per file as opposed to eight?
Alter the line as follows:
group_concat(DISTINCT tf.id) as `tfid`,
This then only gets you the unique ids.
If you want them ordered add:
group_concat(DISTINCT tf.id ORDER BY tf.id ASC) as `tfid`,
I have 2 tables : users and paypal_transactions
For each user, we have an id (named user_id in paypal_transactions table)
A user may have several paypal_transactions. Relation one to many
I need to grab the latest transaction id (ordered by date_dt DESC) when I do my query
My current query :
SELECT `Transaction`.*, `User`.*, `Tipster`.`username`
FROM `pronostics_framework`.`users` AS `User`
LEFT JOIN `pronostics_framework`.`users` AS `Tipster` ON (`User`.`tipster_id` = `Tipster`.`id`)
LEFT JOIN `pronostics_framework`.`paypal_transactions` AS `Transaction` ON (`User`.`id` = `Transaction`.`user_id`)
ORDER BY `User`.`id` DESC
LIMIT 500
Currently with one transaction per user, it works fine. BTW with many transactions I still get the 1st entry from paypal_transactions table (the oldest, but I want the latest from now).
I did many tries, without success.
Thanks for your help !
Here you go:
SELECT `Transaction`.*, `User`.*, `Tipster`.`username`
FROM `pronostics_framework`.`users` AS `User`
LEFT JOIN `pronostics_framework`.`users` AS `Tipster` ON (`User`.`tipster_id` = `Tipster`.`id`)
LEFT JOIN (SELECT user_id, MAX(date_dt) AS max_date
FROM `pronostics_framework`.paypal_transactions
GROUP BY user_id) AS max_trans
ON User.id = max_trans.user_id
LEFT JOIN `pronostics_framework`.`paypal_transactions` AS `Transaction`
ON (max_trans.user_id = `Transaction`.`user_id` AND max_trans.max_date = Transation.date_dt)
ORDER BY `User`.`id` DESC
LIMIT 500
Another way, based on the first query in the first answer at Retrieving the last record in each group:
SELECT `Transaction`.*, `User`.*, `Tipster`.`username`
FROM `pronostics_framework`.`users` AS `User`
LEFT JOIN `pronostics_framework`.`users` AS `Tipster` ON (`User`.`tipster_id` = `Tipster`.`id`)
LEFT JOIN `pronostics_framework`.`paypal_transactions` AS `Transaction` ON Transaction.user_id = User.id
LEFT JOIN `pronostics_framework`.paypal_transactions AS Transactions1
ON Transactions1.user_id = Transactions.user_id AND Transactions1.user_id > Transactions.user_id
WHERE Transactions1.user_id IS NULL
ORDER BY `User`.`id` DESC
LIMIT 500
I am running a MySQL query to get all "users" with current orders.
(It is possible for a user to have more than 1 associated orders in the db/query).
However i also want to get the total order value for each user and total order count for each user that is being returned (within the below query).
I could do these calculations in PHP, but feel it is possible and would be neater all done within the same SQL query (if possible).
This is the basic query with no attempt to make the above calculations
SELECT u.UserID, u.UserName,
o.OrdersID, o.OrderProductName, o.OrderProductQT, o.OrderTotalPrice, o.tUsers_UserID, o.tOrderStatus_StatusID, o.OrderDate, o.OrderDateModified, o.OrderVoid, o.tProducts_ProductID,
os.OrderStatusName,
ud.UserDetailsName, ud.UserDetailsPostCode,
p.ProductName, p.ProductImage1
FROM tusers u
INNER JOIN torders o ON o.tUsers_UserID = u.UserID
INNER JOIN torderstatus os ON os.OrderStatus_StatusID = o.tOrderStatus_StatusID
INNER JOIN tuserdetails ud ON ud.tUsers_UserID = u.UserID
LEFT JOIN tproducts p ON p.ProductID = o.tProducts_ProductID
WHERE o.tOrderStatus_StatusID = ?
GROUP BY u.UserID
ORDER BY OrdersID DESC
I have tried various nested select queries, but none of them work (right)
Is what i want to do possible in SQL or should i just do it all in PHP once i have the returned query results?
Any advice is much appreciated
You can embed the slightly modified queries into another query. For instance:
SELECT userid, SUM(orderid) FROM orders GROUP BY userid
and
SELECT userid, SUM(distinct productid)
FROM
orders o
INNER JOIN orderlines ol on ol.orderid = o.orderid
GROUP BY
userid
can be combined to:
SELECT
u.userid
u.fullname,
(SELECT SUM(orderid)
FROM orders o
WHERE o.userid = u.userid) as ORDERCOUNT,
(SELECT SUM(distinct productid)
FROM
orders o
INNER JOIN orderlines ol on ol.orderid = o.orderid
WHERE
o.userid = u.userid) as UNIQUEPRODUCTS
FROM
users u
Note that the latter query will return all users and will return NULL for ORDERCOUNT or UNIQUEPRODUCTS when the subquery doesn't return anything (when a user doesn't have orders). Also, the query will fail when a subquery returns more than 1 row, which should never happen in the example I posted.
I currently have:
SELECT tbl_review.*, users.first_name, users.last_name, (
SELECT order_ns.tran_date
FROM order_ns
LEFT JOIN product_2_order_ns.external_order_id = order_ns.order_id
WHERE product_2_order_ns.bkfno IN :id
ORDER BY order_ns.trandate ASC
LIMIT 1
) as purchase_date
FROM tbl_review
LEFT JOIN users ON users.sequal_user_id = tbl_review.user_id
WHERE tbl_review.product_id IN :id AND tbl_review.approved = 1
Which, in its sub query, selects an order the user has which has a product in question (defined in :id) get the the oldest transaction date on file for one of the found orders.
I would really like to keep this to one call of the database (don't really want to call again for each returned user for just one field, or even do a range query of all users) but obviously this particular query isn't working.
What can I do, if anything, to get this working?
I cannot make the sub query into a join since they are two distinct pieces of data, the sub query needs to return detail for each row in the main query.
I think you just want a correlated subquery. It is unclear exactly what the relationship is between the inner query and the outer one. My guess is that it is on users and orders:
SELECT tbl_review.*, users.first_name, users.last_name,
(SELECT order_ns.tran_date
FROM order_ns LEFT JOIN
product_2_order_ns
on product_2_order_ns.external_order_id = order_ns.order_id and
product_2_order_ns.bkfno = tbl_review.product_id and
WHERE order_ns.user_id = tbl_review.user_id
ORDER BY order_ns.trandate ASC
LIMIT 1
) as purchase_date
FROM tbl_review LEFT JOIN
users
ON users.sequal_user_id = tbl_review.user_id
WHERE tbl_review.product_id IN :id AND tbl_review.approved = 1;
EDIT:
Oh, the inner query has no relationship to the outer query. Then it is easier. Move it to the from clause using cross join:
SELECT tbl_review.*, users.first_name, users.last_name,
innerquery.tran_date as purchase_date
FROM tbl_review LEFT JOIN
users
ON users.sequal_user_id = tbl_review.user_id cross join
(SELECT order_ns.tran_date
FROM order_ns LEFT JOIN
product_2_order_ns
on product_2_order_ns.external_order_id = order_ns.order_id
WHERE product_2_order_ns.bkfno IN :id
ORDER BY order_ns.trandate ASC
LIMIT 1
) innerquery
WHERE tbl_review.product_id IN :id AND tbl_review.approved = 1;
#Gordons answer is really close but I wanted it to return even if no data was found for tran_date so I changed my query to:
SELECT tbl_review.*, users.first_name, users.last_name, order_ns.tran_date
FROM tbl_review
LEFT JOIN users ON users.sequal_user_id = tbl_review.user_id
LEFT JOIN order_ns ON order_ns.order_id = (
SELECT order_ns.order_id
FROM order_ns
LEFT JOIN product_2_order_ns on product_2_order_ns.external_order_id = order_ns.order_id
WHERE product_2_order_ns.bkfno IN :id
ORDER BY order_ns.tran_date ASC
LIMIT 1
)
WHERE tbl_review.product_id IN :id AND tbl_review.approved = 1;
This will return the distinct data of tran_date irrespective of whether it is found or not.