I'm working on an application with has millions of data, and I need to list all the data in a page. At first I was fetching all the data in a single query and the pagination was done in the client side. But that takes almost 15 minutes to load the complete set. So I changed the code for fetching 10 rows per request and the pagination was done on server side. But still the performance is not up to the mark. So what all things should do to get the data quickly or what is the best way to handle huge data.
My query to fetch data :
UPDATED :
SELECT w.work_order_id,
(SELECT CONCAT(user_fname, ' ', user_lname) FROM users WHERE user_id = w.created_by) AS created_by,
CASE w.assignee WHEN 0 THEN 'All' WHEN -1 THEN 'Unassigned' ELSE CONCAT(u.user_fname, ' ', u.user_lname) END AS assignee
FROM FiveVan_work_orders w
LEFT JOIN users u ON (u.user_id = w.assignee)
WHERE ( w.work_order_status != 'Deleted' && w.work_order_status != 'Closed') ORDER BY w.created_on DESC LIMIT 0,10;
I have created index for the pages and this is the result of explaining the query
+----+--------------------+-------+--------+-------------------+---------------+---------+-------------------------------+--------+------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------+--------+-------------------+---------------+---------+-------------------------------+--------+------------------------------------------+
| 1 | PRIMARY | w | index | work_order_status | work_order_id | 790 | NULL | 340319 | Using where; Using index; Using filesort |
| 1 | PRIMARY | u | eq_ref | PRIMARY | PRIMARY | 4 | fivevan_loadtest.w.assignee | 1 | NULL |
| 2 | DEPENDENT SUBQUERY | users | eq_ref | PRIMARY | PRIMARY | 4 | fivevan_loadtest.w.created_by | 1 | NULL |
+----+--------------------+-------+--------+-------------------+---------------+---------+-------------------------------+--------+------------------------------------------+
WHERE ( w.work_order_status != 'Deleted' &&
w.work_order_status != 'Closed')
ORDER BY w.created_on DESC
If there is only one more value of work_order_status, then change that to
WHERE w.work_order_status = 'Open'
ORDER BY w.created_on DESC
and add
INDEX(work_order_status, created_on)
If there are multiple other values, the this may work (less well):
INDEX(created_on)
To get even better performance, you need to "remember where you left off" instead of using OFFSET. I discuss that in my pagination blog.
Related
Hi I have table threads and questions with relation one-to-many.
every message has foreign key thread_id.
I want to check is last message in the thread is from user and if yes then count it, so it will count all unanswered threads by admin.
+----+-------------+---------+----------+------+
| id | thread_id | user_id | admin_id | text |
| 1 | 1 | 1 | NULL | test |
| 2 | 1 | 1 | NULL | test |
| 3 | 1 | 1 | NULL | test |
| 4 | 1 | 1 | NULL | test |
+----+-------------+---------+----------+------+
The result from this dataset should be 1, because all mesages belongs to the same thread and there is no admin answer (last row has admin_id = NULL).
I tried this:
ThreadMessage::where('admin_id', '=', null)->groupBy('thread_id')->orderBy('id', 'DESC')->limit(1)->count();
Sadly this just counts all the messages where admin_id is null, not just the last ones in thread.
The answer can be in mysql OR laravel eloquent. Thank you very much.
To check for null you need to use IS NULL not = NULL operator. Use whereNull() method
ThreadMessage::whereNull('admin_id')->groupBy('thread_id')->count();
EDIT
To check if the last id is reponded to by admin, you can do it in two ways
with code
$thread = ThreadMessage::orderBy('thread_id', 'desc')->groupBy('thread_id')->first();
$answeredByAdmin = $thread->admin_id != null;
With query (assuming the table name is thread_messages)
ThreadMessage:::whereNull('admin_id')
->groupBy('thread_id')
->whereIn('id', \DB::raw('(SELECT MAX(id) FROM thread_messages GROUP BY question_id)'))
->count();
Title says it all. Do you think anyone could help me untangle this? or if someone could point me out to what else could be causing it to take so much time. The query takes about half an hour to run. The guy who wrote this tried doing it in a loop, by removing the table from the last join statement and then querying the field.title for each vote. i was hoping to bring the result to about 5 mins.
some extra info:
The query result is 83,531 rows
The vote table size is 30 MB (261,169 rows)
SELECT `vote`.`id` `vote_id`, `branch`.`name` `branch`, `brand`.`name` `brand`, DATE(vote.created_at) `date`, HOUR(vote.created_at) `time_hour`,
MINUTE(vote.created_at) `time_minute`, `vote`.`is_like`, `voter`.`name`, `voter`.`telephone`, `voter`.`email`, popups_votes.title `popup_title`,
popups_votes.value `popup_value`, GROUP_CONCAT(dis.field SEPARATOR '|') `reasons`
FROM (`vote`)
LEFT JOIN `voter` ON `voter`.`id` = `vote`.`voter_id`
LEFT JOIN `device` ON `device`.`id` = `vote`.`device_id`
LEFT JOIN `branch` ON `branch`.`id` = `device`.`branch_id`
LEFT JOIN `brand` ON `brand`.`id` = `branch`.`brand_id`
LEFT JOIN `popups_votes` ON popups_votes.vote_id = vote.id
LEFT JOIN (SELECT vote_dislike.vote_id `vote_id`, field.title `field` FROM vote_dislike
LEFT JOIN branch_dislike_field ON branch_dislike_field.id = vote_dislike.branch_dislike_id
LEFT JOIN field ON field.id = branch_dislike_field.field_id) dis
ON dis.vote_id = vote.id
WHERE (vote.device_id in
(
Select d.id
From device d
WHERE d.branch_id IN (SELECT id FROM branch WHERE brand_id = 7)
)
)
AND (vote.created_at >= FROM_UNIXTIME('$from_time') AND vote.created_at <= FROM_UNIXTIME('$to_time') )
GROUP BY vote.id
EDIT: this is the explain {query} output:
+------+-------------+----------------------+--------+----------------------+-----------+---------+-------------------------------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+----------------------+--------+----------------------+-----------+---------+-------------------------------------------+------+----------------------------------------------+
| 1 | PRIMARY | branch | ref | PRIMARY,brand_id | brand_id | 4 | const | 20 | Using index; Using temporary; Using filesort |
| 1 | PRIMARY | d | ref | PRIMARY,branch_id | branch_id | 4 | river_back.branch.id | 1 | Using index |
| 1 | PRIMARY | vote | ref | device_id,created_at | device_id | 4 | river_back.d.id | 1200 | Using where |
| 1 | PRIMARY | voter | eq_ref | PRIMARY | PRIMARY | 4 | river_back.vote.voter_id | 1 | |
| 1 | PRIMARY | device | eq_ref | PRIMARY | PRIMARY | 4 | river_back.d.id | 1 | |
| 1 | PRIMARY | branch | eq_ref | PRIMARY | PRIMARY | 4 | river_back.device.branch_id | 1 | Using where |
| 1 | PRIMARY | brand | eq_ref | PRIMARY | PRIMARY | 4 | river_back.branch.brand_id | 1 | Using where |
| 1 | PRIMARY | popups_votes | ref | vote_id | vote_id | 5 | river_back.vote.id | 602 | |
| 1 | PRIMARY | vote_dislike | ref | vote_id | vote_id | 4 | river_back.vote.id | 1 | |
| 1 | PRIMARY | branch_dislike_field | eq_ref | PRIMARY | PRIMARY | 4 | river_back.vote_dislike.branch_dislike_id | 1 | Using where |
| 1 | PRIMARY | field | eq_ref | PRIMARY | PRIMARY | 4 | river_back.branch_dislike_field.field_id | 1 | Using where |
+------+-------------+----------------------+--------+----------------------+-----------+---------+-------------------------------------------+------+----------------------------------------------+
You should check that all the data you are selecting are indexed and you have foreign keys.
How do MySQL indexes work?
Basically an index on a table works like an index in a book (that's
where the name came from):
Let's say you have a book about databases and you want to find some
information about, say, storage. Without an index (assuming no other
aid, such as a table of contents) you'd have to go through the pages
one by one, until you found the topic (that's a full table scan). On
the other hand, an index has a list of keywords, so you'd consult the
index and see that storage is mentioned on pages 113-120,231 and 354.
Then you could flip to those pages directly, without searching (that's
a search with an index, somewhat faster).
Basics of Foreign Keys in MySQL?
FOREIGN KEYS just ensure your data are consistent.
They do not improve queries in sense of efficiency, they just make
some wrong queries fail.
Do not use LEFT unless you are expecting the "right" table to have missing rows.
In particular, the Optimizer probably cannot start with the 'derived table' since it is hiding to the right of a LEFT.
Do not use IN ( SELECT ... ); if possible change to EXISTS ( SELECT * ...) or JOIN.
Try to avoid the "inflate-deflate" caused by JOIN ... GROUP BY. If possible find the ids of interest without needing a GROUP BY, then JOIN to the other tables.
Putting many of those together, does this get you close to the desired result, at least in the sense of getting the correct vote.id values?
SELECT vote.id
FROM vote AS v
JOIN (
SELECT vote_dislike.vote_id `vote_id`, field.title `field`
FROM vote_dislike AS vd
LEFT JOIN branch_dislike_field AS bd
ON bd.id = vd.branch_dislike_id
LEFT JOIN field
ON field.id = bd.field_id
) AS dis ON dis.vote_id = v.id
JOIN device AS d ON v.device_id = d.id
JOIN branch AS b ON d.branch_id = b.id
WHERE b.brand_id = 7
AND v.created_at >= ...
AND v.created_at <= ...
Then:
SELECT lots of stuff
FROM ( the above query ) AS x
JOIN vote v ON x.id = v.id -- yes, dig back into `vote` for the other stuff
JOIN voter ...
JOIN ...
but with no GROUP BY.
Currently we are using a custom CI library to generate PDF files from documents which exist as database records in our database.
Each document is related to the contents (== rows) with a one-has-many relation. Each row has a number (field: row_span) to indicate how many lines it will use once it gets printed in the PDF.
Per PDF page that gets build, Rows needed for that page only are selected using a subquery:
$where = $this->docType."_id = ".$this->data['doc']->id." AND visible = 1";
$sql = "SELECT *,
(SELECT
sum(row_span) FROM app_".$this->docType."_rows X
WHERE X.position <= O.position
AND ".$where."
ORDER BY position ASC) 'span_total'
FROM app_".$this->docType."_rows O
WHERE ".$where."
HAVING span_total > ".(($i-1)*$this->maxRows)." AND span_total <= ".($i*$this->maxRows)." ORDER BY O.position ASC ";
$rows = $rows->query($sql);
In the code $i is the page number and $this->maxRows is loaded from the document template record which indicates how many available lines the PDF template has.
So when the SQL renders it might look like this for page 1 of an order with ID 834:
SELECT `app_order_rows`.*,
(SELECT SUM(`app_order_rows_subquery`.`row_span`) AS row_span
FROM `app_order_rows` `app_order_rows_subquery`
WHERE `app_order_rows_subquery`.`position` <= 'app_order_rows.position'
AND `app_order_rows_subquery`.`order_id` = 834
AND `app_order_rows_subquery`.`visible` = 1
ORDER BY `app_order_rows_subquery`.`position` asc) AS span_total
FROM (`app_order_rows`)
WHERE `app_order_rows`.`order_id` = 834
AND `app_order_rows`.`visible` = 1
HAVING span_total > 0
AND span_total <= 45
ORDER BY `app_order_rows`.`position` asc
And running this with EXPLAIN gives this as output:
+====+=============+=========================+======+===============+======+=========+======+======+=============================+===+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | |
+====+=============+=========================+======+===============+======+=========+======+======+=============================+===+
| 1 | PRIMARY | app_order_rows | ALL | NULL | NULL | NULL | NULL | 1809 | Using where; Using filesort | 1 |
+----+-------------+-------------------------+------+---------------+------+---------+------+------+-----------------------------+---+
| 2 | SUBQUERY | app_order_rows_subquery | ALL | NULL | NULL | NULL | NULL | 1809 | Using where | 2 |
+====+=============+=========================+======+===============+======+=========+======+======+=============================+===+
This is working great, but... When we have large orders or invoices it renders the documents very slow. This might be due to the subquery.
Does anyone have an idea on how to do the same select without subquery? Maybe we will have to go for a whole new approach to select rows and build the PDF. We are open for suggestions ^^
Thanks in advance
------------------------------- edit ------------------------------
The EXPLAIN after index creation:
+====+=============+=========================+=======+===============+============+=========+=======+======+=============+===+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | |
+====+=============+=========================+=======+===============+============+=========+=======+======+=============+===+
| 1 | PRIMARY | app_order_rows | ref | index_main | index_main | 5 | const | 9 | Using where | 1 |
+----+-------------+-------------------------+-------+---------------+------------+---------+-------+------+-------------+---+
| 2 | SUBQUERY | app_order_rows_subquery | range | index_main | index_main | 10 | NULL | 1 | Using where | 2 |
+====+=============+=========================+=======+===============+============+=========+=======+======+=============+===+
As you confirmed in the comments, the tables have no indexes.
The immediate solution would be:
create index index_main on app_order_rows (order_id, position);
Why do this two queries return different result set when they have the same ORDER BY.
Only difference in query is that first time I user INNER JOIN an it takes about 5 seconds.
Second time I used LEFT JOIN and it took 0.05 seconds. In both cases they return exactly 43.000 rows, but tck.id order is different and I can't figure out why or in which way?
SELECT tck.*, acc.ac_name
FROM support_tickets tck
INNER JOIN support_ticket_accounts acc USING (id_support_ticket_account)
WHERE tck.id_company = 2 AND tck.st_status = 1 ORDER BY tck.st_priority DESC
Edit:
SELECT tck.*, acc.ac_name
FROM support_tickets tck
LEFT JOIN support_ticket_accounts acc ON tck.id_support_ticket_account = acc.id_support_ticket_account
WHERE tck.id_company = 2 AND tck.st_status = 1
ORDER BY tck.st_priority DESC;
+----+-------------+-------+--------+---------------+------------+---------+-------------------------------+-------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+------------+---------+-------------------------------+-------+-----------------------------+
| 1 | SIMPLE | tck | ref | id_company | id_company | 5 | const | 37586 | Using where; Using filesort |
| 1 | SIMPLE | acc | eq_ref | PRIMARY | PRIMARY | 4 | tck.id_support_ticket_account | 1 | |
+----+-------------+-------+--------+---------------+------------+---------+-------------------------------+-------+-----------------------------+
SELECT tck.*, acc.ac_name
FROM support_tickets tck
INNER JOIN support_ticket_accounts acc ON tck.id_support_ticket_account = acc.id_support_ticket_account
WHERE tck.id_company = 2 AND tck.st_status = 1
ORDER BY tck.st_priority DESC;
+----+-------------+-------+------+-------------------------------------+----------------------------+---------+-------------------------------+------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+--------------------------------------+---------------------------+---------+-------------------------------+------+---------------------------------+
| 1 | SIMPLE | acc | ALL | PRIMARY | NULL | NULL | NULL | 5 | Using temporary; Using filesort |
| 1 | SIMPLE | tck | ref | id_company,id_support_ticket_account | id_support_ticket_account | 5 | acc.id_support_ticket_account | 2085 | Using where |
+----+-------------+-------+------+--------------------------------------+---------------------------+---------+-------------------------------+------+---------------------------------+
I think using temporary is responsible for the delay (but I don't see why it's necessary for one query and not the other one). I think creating multi-column index should help:
CREATE INDEX filter
ON support_tickets(id_company, st_status, st_priority)
USING BTREE;
If you just ORDER BY tck.st_priority DESC multiple different recordsets are posible and can be returned, for each of both cases (left or inner). That is because you must have a lot of records that has the same st_priority so any of them can came in no particular order
Add more fields to the order by clause to give any record unique possible position and you will have same order on both querys.
is there any way to remove the MAX with something else, maybe the ASC LIMIT 1 to decrease database throttling? My query gets the maximum id and adds 1 to it.
This is my query:
$query = 'SELECT MAX(ID) +1 as maxidpost
FROM wp_posts';
$result = mysql_query($query) or die(mysql_error());
while($row = mysql_fetch_array($result)) {
echo 'p='. $row['maxidpost'];
}
mysql_close();
What does the database tell you about your query? If id is indexed then
mysql> explain select max(id) + 1 from times;
+----+-------------+-------+------+---------------+------+---------+------+------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+------+------------------------------+
| 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away |
+----+-------------+-------+------+---------------+------+---------+------+------+------------------------------+
1 row in set (0.18 sec)
mysql> explain select id from times order by id ASC limit 1;
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | times | index | NULL | PRIMARY | 4 | NULL | 1 | Using index |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------------+
1 row in set (0.00 sec)
This is fine. adding limit 1 isn't going to give any noticeable performance improvements.
This shouldn't cause any kind of issues, are you having problems with this particular query?
The SQL MAX() function will automatically look for just one result. There's no need of LIMIT it in any way. If you want to improve that query try setting ID as index.
SELECT ID + as maxidpost FROM wp_posts ORDER BY ID DESC LIMIT 1;
or If the table have Auto_increment ID
SHOW TABLE STATUS LIKE 'wp_posts';
There should be a field called Auto_increment which should be exactly MAX(ID) + 1;