We are running OpenCart store and now we have to export order data for 6 months (around 17K orders).
Opencart has a builtin solution, when you select the orders you need and export, but it works fine with less than 500 orders.
I've decided to make a standalone script based on the query it uses for its original export.
I'm not publishing the export part, since my dedicated server and the store freezes when I run the following query and consequent loop for mysql_fetch_assoc.
$query = "SELECT o.*, op.name, op.model,op.order_product_id, op.product_id, op.quantity, op.price, op.total AS ptotal, op.tax,
(SELECT ot.value FROM order_total ot WHERE ot.order_id = o.order_id AND ot.code = 'sub_total') AS sub_total,
(SELECT ot.value FROM order_total ot WHERE ot.order_id = o.order_id AND ot.code = 'credit') AS store_credit,
(SELECT `name` FROM order_history oh INNER JOIN `order_status` os on oh.order_status_id=os.order_status_id WHERE oh.order_id = o.order_id and os.language_id='2'
ORDER BY order_history_id DESC LIMIT 1) as order_status
FROM `order` o inner join `order_product` op on o.order_id=op.order_id WHERE o.date_added BETWEEN '2014-01-01 00:00:00' AND '2014-06-30 03:59:59'";
$sql = mysql_query($query);
do {
echo $o['model']."<br/>"; // here we will have a part using PEAR Excel basically
} while ($o=mysql_fetch_assoc($sql));
$endtime = microtime();
echo $endtime-$starttime;
Indexes are set. Additional index was set on oh.order_id. No luck. The same request for 1 particular order id runs in 0.003ms.
My process list shows "Sending data" for the above shown query and "Waiting for table level lock" for anothers.
Could you kindly assist?
I think you could optimize your SQL a bit to prevent locks. The following might help, but recognize that I don't have your schema or data to test this on, so you may need to tweak.
SELECT o.*, op.name, op.model,op.order_product_id, op.product_id,
op.quantity, op.price, op.total AS ptotal, op.tax,
ot.sub_total, ot.store_credit, os.os_name
FROM `order` o inner join `order_product` op on o.order_id=op.order_id
INNER JOIN (SELECT t.order_id,
MAX(case when t.code = 'sub_total' t.value else 0 end case) as sub_total,
MAX(case when t.code = 'credit' t.value else 0 end case) as store_credit
FROM order_total t GROUP BY t.order_id) ot ON ot.order_id = o.order_id
INNER JOIN (SELECT * FROM (SELECT oh.order_id, `name` AS os_name
FROM order_history oh
INNER JOIN `order_status` s on oh.order_status_id=s.order_status_id
WHERE s.language_id='2' ORDER BY order_history_id DESC) t1
GROUP BY t1.order_id) as os ON os.order_id = o.order_id
WHERE o.date_added BETWEEN '2014-01-01 00:00:00' AND '2014-06-30 03:59:59'
I would also probably do a create temp table temp_order_history from select ... and then select * from temp_order_history Also, you may need to do outer join instead of inner join if either of those sub selects have missing data. In those cases, you'd just get NULLs for those columns.
Related
I have this query
select courses.id, y.qs_world, courses.name_en as name,
courses.description_en as description,
source_link, courses.slug, fee, duration, courses.university_id, college_id,
study_level_id, application_fee, courses.currency_id
from courses
left join university_ranks as y on courses.university_id = y.university_id
and y.year = '2021'
left join universities on courses.university_id = universities.id
left join countries on countries.id = universities.country_id where courses.status = 1
order by ROW_NUMBER() OVER (PARTITION BY countries.id ORDER BY courses.id)
This query is taking too long to execute, but it is working well if I remove the last row.
I used indexing but nothing different.
The EXPLAIN notes are to Using temporary,Using filesort but I want to improve the query without using temporary or filesort
How can I achieve this?
UPDATE:
I tried this query but same speed
SELECT * FROM (
SELECT
`courses`.`id`,`courses`.`status`, `y`.`qs_world`, `courses`.`name_en` as `name`, `courses`.`description_en` as `description`,
`source_link`, `courses`.`slug`, `fee`, `duration`, `courses`.`university_id`, `college_id`,
`study_level_id`, `application_fee`, `courses`.`currency_id`, `countries`.`id` as country_id
FROM
courses
left join `university_ranks` as `y` on `courses`.`university_id` = `y`.`university_id`
and `y`.`year` = '2021'
left join `universities` on `courses`.`university_id` = `universities`.`id`
left join `countries` on `countries`.`id` = `universities`.`country_id`
) UserCourse where status = 1
order by ROW_NUMBER() OVER (PARTITION BY country_id ORDER BY id)
countries.id as country_id --> universities.country_id
then remove
left join `countries` ON `countries`.`id` = `universities`.`country_id`
Move where status = 1 into the inner query.
It seems like
order by ROW_NUMBER() OVER (PARTITION BY country_id ORDER BY id)
could be replaced by
ORDER BY country_id, id
Get rid of the outer query
Don't say LEFT unless the 'right' table might have a missing row. (It confuses the reader as to your intent.)
I'm trying to speed up the following query as it takes quite long to run: now it's 'only' about 1.5 seconds, but it will certainly get slower with more rows (which will 10x over the next period).
Basically, I want to show all the rows from the orders table for the user, and per row show the total order amount (which is the SUM of the orders_products table).
SELECT
orders.order_number,
orders.order_date,
companies.company_name,
COALESCE(SUM(orders_products.product_price * orders_products.product_quantity),0) AS order_value
FROM orders
LEFT JOIN companies ON companies.id = orders.company_id
LEFT JOIN orders_products ON orders_products.order_id = orders.id
LEFT JOIN users ON users.id = orders.user_id
WHERE orders.user_id = '$user_id'
AND companies.user_id = '$user_id'
GROUP BY orders.id ORDER BY orders.order_date DESC, orders.order_number DESC
I've tried adding another condition AND orders_products.user_id = '$user_id'. Speed wise the query was about 12x faster (yeah!) but the problem is that not all orders have products in them. In this case, the orders without products in them are not returned.
How do I change my query so that despite of an order not having products in them, it still is returned (with total order value 0), whilst also speeding up the query?
Thank you in advance for your help!
You might find it faster to use a correlated subquery:
SELECT o.order_number, o.order_date, c.company_name,
(SELECT COALESCE(SUM(op.product_price * op.product_quantity), 0)
FROM orders_products op
WHERE op.order_id = o.id
) AS order_value
FROM orders o LEFT JOIN
companies c
ON c.id = o.company_id AND c.user_id = o.user_id
WHERE o.user_id = '$user_id'
ORDER BY o.order_date DESC, o.order_number DESC
This gets rid of the outer aggregation which is often a performance win.
Then for performance you want the following indexes:
orders(user_id, order_date desc, order_number_desc, company_id)
companies(id, company_id, company_name)
orders_products(order_id, product_price, product_quantity)
I have this query that will be used for reports and i have a lot of data.
As of the moment is this my query that uses of UNION ALL over UNION because i remembered last project i used the UNION is that it dont unite all the datas and i never knew why so i used the UNION ALL
SELECT `item_id`, NULL AS `ob`,SUM(`add`) AS `add` ,NULL AS `dam`,NULL AS `sold`,NULL AS `others`,NULL AS `ac`
FROM (
-- DELIVERED ITEMS
SELECT si.`item_id`,SUM(sdi.`actual_pcs`) AS `add`
FROM `store_inventory` si
LEFT JOIN `store_delivery` sd
ON sd.`store_id` = si.`store_id` AND DATE(sd.`dt_modified`) = '2018-09-01'
AND sd.`status` = "DONE"
LEFT JOIN `store_delivery_items` sdi
ON sdi.`strd_id` = sd.`strd_id` AND sdi.`item_id` = si.`item_id`
WHERE si.`store_id` = '4'
GROUP BY si.`item_id`
UNION ALL
-- OVERRIDE ADD (ADD)
SELECT si.`item_id`,SUM(oi.`qty`) AS `add`
FROM `store_inventory` si
LEFT JOIN `override` o
ON o.`ref_id` = si.`store_id`
AND o.`type` = "STORE"
AND o.`action` = "ADD"
AND DATE(o.`dt_created`) = '2018-09-01'
LEFT JOIN `override_items` oi
ON oi.`item_id` = si.`item_id`
AND oi.`ovr_id` = o.`ovr_id`
WHERE si.`store_id` = '4'
GROUP BY si.`item_id`
UNION ALL
-- CANCELLED SALES (ADD)
SELECT si.`item_id`,SUM(sai.`si_qty`) AS `add`
FROM `store_inventory` si
LEFT JOIN `sales` s
ON s.`sales_store_id` = si.`store_id`
AND s.`cancelled_by` IS NOT NULL
AND DATE(s.`dt_cancelled`) = '2018-09-01'
LEFT JOIN `sales_items` sai
ON sai.`si_sales_id` = s.`sales_id`
AND sai.`si_item_id` = si.`item_id`
WHERE si.`store_id` = '4'
GROUP BY si.`item_id`
UNION ALL
-- PULLOUT REJECTED ADDED BACK TO STORE (ADD)
SELECT si.`item_id`,SUM(poi.`poi_actual`) AS `add`
FROM `store_inventory` si
LEFT JOIN `pullout` p
ON p.`pullout_reason` IN ("STORE TO DAMAGE","STORE TO WAREHOUSE")
AND p.`pullout_status` IN ("REJECT","HEAD-REJECTED")
AND DATE(p.`dt_modified`) = '2018-09-01'
AND p.`pullout_from` = si.`store_id`
LEFT JOIN `pullout_items` poi
ON poi.`poi_item_id` = si.`item_id`
AND poi.`poi_pullout_id` = p.`pullout_id`
WHERE si.`store_id` = '4'
GROUP BY si.`item_id`
) AS add_temp
GROUP BY `item_id`
As you can see i loop all my table from my store inventory items this is only a one day report because i do the loop in php for date ranges. Im wondering if this query can be enhanced more. I tried the left join all over my tables and i got no luck. Is this the right query for this kind of reporting? this is just part my query BTW. I still have this openingbalance,damaged,sold,others query as you can see in my SELECT statement. So i really need an advise regarding with this long query. I already added indeces of my columns to my tables in order to select results faster.
In my query I am listing all of the theater ticket sales and movie ticket sales of different customers. The issue I'm running into is that all of the '0' ticket sales, so those users who haven't boughten a theater ticket or movie ticket is not showing up.
Here's a picture for a visual aspect: table
I believe I need to be doing a union to return the users who haven't boughten any tickets. I just can't seem to figure this out.
Thanks in advance.
Here's my code so far:
select customer.hippcode, customer.LastName, customer.Firstname, customer.Email,
count(ticketdetails.eventtype) as 'Theater Tickets',
0 as 'Movie Tickets'
from customer
inner join ticketdetails on ticketdetails.hippcode = customer.hippcode
where ticketdetails.hippcode is not null
and ticketdetails.eventType ='T'
Group by Customer.hippcode
union
select customer.hippcode, customer.LastName, customer.Firstname, customer.Email,
0 as 'Theater Tickets', count(ticketdetails.eventtype) as 'Movie Tickets'
from customer
inner join ticketdetails on ticketdetails.hippcode = customer.hippcode
where ticketdetails.hippcode is not null
and ticketdetails.eventType ='M'
Group by Customer.hippcode
order by `theater tickets` + `movie tickets` desc;
select
customer.hippcode, customer.LastName, customer.Firstname, customer.Email,
sum(case when ticketdetails.eventtype = 'T' then 1 else 0 end) as TheaterTickets,
sum(case when ticketdetails.eventtype = 'M' then 1 else 0 end) as MovieTickets
from customer
inner join ticketdetails on ticketdetails.hippcode = customer.hippcode
where ticketdetails.hippcode is not null
and ticketdetails.eventType in ('T', 'M')
Group by customer.hippcode, customer.LastName, customer.Firstname, customer.Email
Order by 'TheaterTickets' + 'MovieTickets' desc
inner join => bring the line only if you got a record on both table.
I think you should use a LEFT JOIN somewhere to chose the master table
http://dev.mysql.com/doc/refman/5.7/en/join.html and
http://dev.mysql.com/doc/refman/5.7/en/left-join-optimization.html
I think the last query is the only one you want. A left join is appropriate, but you need to be careful about the where clause:
select c.hippcode, c.LastName, c.Firstname, c.Email,
sum(td.eventtype) as TheaterTickets,
sum(td.eventtype) as MovieTickets
from customer c left join
ticketdetails td
on td.hippcode = c.hippcode and
td.eventType in ('T', 'M')
Group by c.hippcode, c.LastName, c.Firstname, c.Email
Order by count(t.hippcode) desc;
Notes:
Table aliases make the query easier to read and write.
Conditions on ticketdetails go in the on clause, not the where clause.
The condition on td.hippcode is not null is unnecessary, because NULL will not match in the join (note: you might want to check for the customer column).
case is the standard way to do the conditional sum (and hence correct). However, MySQL offers a much simpler and intuitive syntax.
Your order by was doing nothing, because it was adding two strings (hence equivalent to order by 0. Don't use single quotes ever for column names, and you won't have a problem like that.
I build a query a month ago on a website. It was working fine. But after a month I was informed that the website become very slow to load the page.
When I search for the problem, I found that my query is executing very slow to fetch the data from mysql database. Then I check for the database and found that the 4 tables which I was using by joins, have around 216850, 167634, 64000, 931 rows respectively.
I have already have indexed that tables. So, where I'm lacking. Please help guys.
[Edit]
Table1: user_alert
Records: 216850
DB Type: InnoDB
Indexes: id(primary)
Table2: orders
Records: 167634
DB Type: InnoDB
Indexes: id(primary), order_id, customer_id
Table3: user_registration
Records: 64000 around
DB Type: InnoDB
Indexes: id(primary), email_address
Table4: cities
Records: 931
DB Type: InnoDB
Indexes: id(primary)
Query:
SELECT uas.alert_id, uas.user_id, uas.status, ur.first_name, ur.last_name, ur.email_address, o.order_id,
CASE WHEN ct.city_name IS NULL THEN uas.city_name ELSE ct.city_name END AS city_name
FROM `user_alert` uas
LEFT JOIN orders o ON o.customer_id = uas.user_id
LEFT JOIN user_registration ur ON ur.id = uas.user_id
LEFT JOIN `cities` ct ON ct.city_id = uas.city_id
WHERE uas.status = '1'
GROUP BY uas.user_id
ORDER BY uas.create_date DESC
GROUP BY is used to aggregate values up. For example if you wanted the count of orders by a user you could use COUNT(o.order_id).....GROUP BY uas.user_id. There are multiple orders for each user, but the aggregate function is just counting them here. However if you just select o.order_id when you have a GROUP BY uas.user_id it doesn't know which of the possibly many order_id values to return for that user id.
In this case it possibly doesn't matter as it looks like the order table is the only one where there is multiple rows per use. If you want the latest one you could just use MAX(o.order_id) (assuming that the order_id is assigned is order). But if you wanted the order value it becomes more difficult.
SELECT uas.alert_id, uas.user_id, uas.status, ur.first_name, ur.last_name, ur.email_address, MAX(o.order_id) AS LatestOrderId,
IFNULL(ct.city_name, uas.city_name) AS city_name
FROM `user_alert` uas
LEFT JOIN orders o ON o.customer_id = uas.user_id
LEFT JOIN user_registration ur ON ur.id = uas.user_id
LEFT JOIN `cities` ct ON ct.city_id = uas.city_id
WHERE uas.status = '1'
GROUP BY uas.user_id
ORDER BY uas.create_date DESC
If you wanted the (say) value of the latest order then it becomes more difficult.
SELECT uas.alert_id, uas.user_id, uas.status, ur.first_name, ur.last_name, ur.email_address, Sub1.MaxOrderId AS LatestOrderId, o.order_value
IFNULL(ct.city_name, uas.city_name) AS city_name
FROM `user_alert` uas
LEFT JOIN (SELECT customer_id, MAX(order_id) AS MaxOrderId FROM orders GROUP BY customer_id) Sub1 ON Sub1.customer_id = uas.user_id
LEFT OUTER JOIN orders o ON o.customer_id = Sub1.user_id AND o.order_id = Sub1.MaxOrderId
LEFT JOIN user_registration ur ON ur.id = uas.user_id
LEFT JOIN `cities` ct ON ct.city_id = uas.city_id
WHERE uas.status = '1'
ORDER BY uas.create_date DESC
Or doing a bit of a fiddle based on GROUP_CONCAT
SELECT uas.alert_id, uas.user_id, uas.status, ur.first_name, ur.last_name, ur.email_address,
SUBSTRING_INDEX(GROUP_CONCAT(o.order_id ORDER BY o.order_id DESC), ',', 1) AS LatestOrderId,
SUBSTRING_INDEX(GROUP_CONCAT(o.order_value ORDER BY o.order_id DESC), ',', 1) AS LatestOrderValue,
IFNULL(ct.city_name, uas.city_name) AS city_name
FROM `user_alert` uas
LEFT OUTER JOIN orders o ON o.customer_id = uas.user_id AND o.order_id = Sub1.MaxOrderId
LEFT JOIN user_registration ur ON ur.id = uas.user_id
LEFT JOIN `cities` ct ON ct.city_id = uas.city_id
WHERE uas.status = '1'
GROUP BY uas.user_id
ORDER BY uas.create_date DESC