Currently we are using a custom CI library to generate PDF files from documents which exist as database records in our database.
Each document is related to the contents (== rows) with a one-has-many relation. Each row has a number (field: row_span) to indicate how many lines it will use once it gets printed in the PDF.
Per PDF page that gets build, Rows needed for that page only are selected using a subquery:
$where = $this->docType."_id = ".$this->data['doc']->id." AND visible = 1";
$sql = "SELECT *,
(SELECT
sum(row_span) FROM app_".$this->docType."_rows X
WHERE X.position <= O.position
AND ".$where."
ORDER BY position ASC) 'span_total'
FROM app_".$this->docType."_rows O
WHERE ".$where."
HAVING span_total > ".(($i-1)*$this->maxRows)." AND span_total <= ".($i*$this->maxRows)." ORDER BY O.position ASC ";
$rows = $rows->query($sql);
In the code $i is the page number and $this->maxRows is loaded from the document template record which indicates how many available lines the PDF template has.
So when the SQL renders it might look like this for page 1 of an order with ID 834:
SELECT `app_order_rows`.*,
(SELECT SUM(`app_order_rows_subquery`.`row_span`) AS row_span
FROM `app_order_rows` `app_order_rows_subquery`
WHERE `app_order_rows_subquery`.`position` <= 'app_order_rows.position'
AND `app_order_rows_subquery`.`order_id` = 834
AND `app_order_rows_subquery`.`visible` = 1
ORDER BY `app_order_rows_subquery`.`position` asc) AS span_total
FROM (`app_order_rows`)
WHERE `app_order_rows`.`order_id` = 834
AND `app_order_rows`.`visible` = 1
HAVING span_total > 0
AND span_total <= 45
ORDER BY `app_order_rows`.`position` asc
And running this with EXPLAIN gives this as output:
+====+=============+=========================+======+===============+======+=========+======+======+=============================+===+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | |
+====+=============+=========================+======+===============+======+=========+======+======+=============================+===+
| 1 | PRIMARY | app_order_rows | ALL | NULL | NULL | NULL | NULL | 1809 | Using where; Using filesort | 1 |
+----+-------------+-------------------------+------+---------------+------+---------+------+------+-----------------------------+---+
| 2 | SUBQUERY | app_order_rows_subquery | ALL | NULL | NULL | NULL | NULL | 1809 | Using where | 2 |
+====+=============+=========================+======+===============+======+=========+======+======+=============================+===+
This is working great, but... When we have large orders or invoices it renders the documents very slow. This might be due to the subquery.
Does anyone have an idea on how to do the same select without subquery? Maybe we will have to go for a whole new approach to select rows and build the PDF. We are open for suggestions ^^
Thanks in advance
------------------------------- edit ------------------------------
The EXPLAIN after index creation:
+====+=============+=========================+=======+===============+============+=========+=======+======+=============+===+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | |
+====+=============+=========================+=======+===============+============+=========+=======+======+=============+===+
| 1 | PRIMARY | app_order_rows | ref | index_main | index_main | 5 | const | 9 | Using where | 1 |
+----+-------------+-------------------------+-------+---------------+------------+---------+-------+------+-------------+---+
| 2 | SUBQUERY | app_order_rows_subquery | range | index_main | index_main | 10 | NULL | 1 | Using where | 2 |
+====+=============+=========================+=======+===============+============+=========+=======+======+=============+===+
As you confirmed in the comments, the tables have no indexes.
The immediate solution would be:
create index index_main on app_order_rows (order_id, position);
Related
I installed new version of MariaDB and got problem with sorting php-script:
$this->db->query("
SET #v:=0;
UPDATE `users` AS `c1`
LEFT JOIN (SELECT `id`, (#v:=#v+2) AS `ord2`, `name`, `parent`
FROM `users`
WHERE `parent`='0' ORDER BY `ord` ASC) AS `c2`
ON `c1`.`id` = `c2`.`id`
SET `c1`.`ord` = `c2`.`ord2`
WHERE `c1`.`parent` = '0'
");
In particular script sets order of entries in users table sorting them by ord:
+----+-----+--------------------+--------+
| id | ord | name | parent |
+----+-----+--------------------+--------+
| 2 | 2 | admin | 0 |
| 10 | 5 | manager | 0 |
| 12 | 7 | user | 0 |
| 11 | 9 | dev | 0 |
+----+-----+--------------------+--------+
I'm not familiar with SQL and after long hours of searching and tests I, as it seems to me, found out the SQL-query that doesn't work right:
SELECT `id`, (#v:=#v+2) AS `ord2`, `name`, `parent`
FROM `users`
WHERE `parent`='0' ORDER BY `ord` ASC;
In previous version of MariaDB (5.5.5-10.1.25) the query gives entries sorted by initial order (ord):
+----+-----+--------------------+--------+
| id | ord2| name | parent |
+----+-----+--------------------+--------+
| 2 | 2 | admin | 0 |
| 10 | 4 | manager | 0 |
| 12 | 6 | user | 0 |
| 11 | 8 | dev | 0 |
+----+-----+--------------------+--------+
In new version (10.6.3) result is:
+----+-----+--------------------+--------+
| id | ord2| name | parent |
+----+-----+--------------------+--------+
| 2 | 2 | admin | 0 |
| 10 | 4 | manager | 0 |
| 12 | 8 | user | 0 |
| 11 | 6 | dev | 0 |
+----+-----+--------------------+--------+
I tried to set ORDER BY param ord to name and id, but result was the same. It seems like col ord2 filled with even numbers is always sorted by id in ascending order.
How should I rewrite the query or the script to fix the problem?
In latest MariaDB versions you can use window function row_number in next way:
select
id,
(row_number() over (order by ord))*2 ard2,
name,
parent
from tbl;
MariaDB query test here
I'm running into a little issue concerning a datetime picker and MySQL.
I have a PHP script that should get records before (and including) the selected day. However, the dataset returns empty.
Here's the SQL part of the script:
$sql = 'SELECT `pro_title`,
`pro_part_number`,
`name` AS flag_name,
`old_val`,
`new_val`
FROM `products`
INNER JOIN `flags` ON `id` = `pro_flag_id`
INNER JOIN `flag_history` ON `pro_part_number` = `part`
WHERE `pro_flag_id` IN(:flags)
AND STR_TO_DATE(`ts`, "%y-%m-%d") <= :dateTs;';
I then use PDO to bind the params:
array(
':flags' => implode(',', $flags), # this outputs 1,2,3
':dateTs' => $date # this outputs 2019-04-30
)
I've also tried changing <= to >= to no avail (not that it should work, but thought I'd try).
I've come across a good few SO posts but nothing has actually got me there. Here is the description of the flag_history table (where ts is stored)
MariaDB [mastern]> describe `flag_history`;
+---------+-------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+-------------+------+-----+-------------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| part | varchar(25) | NO | | NULL | |
| ts | datetime | YES | | CURRENT_TIMESTAMP | |
| old_val | int(4) | YES | | NULL | |
| new_val | int(4) | YES | | NULL | |
+---------+-------------+------+-----+-------------------+----------------+
And some example data:
MariaDB [mastern]> select * from `flag_history` order by `id` desc limit 5;
+-------+----------+---------------------+---------+---------+
| id | part | ts | old_val | new_val |
+-------+----------+---------------------+---------+---------+
| 24026 | PART-001 | 2019-04-30 09:42:22 | 0 | 3 |
| 24025 | PART-002 | 2019-04-30 09:42:22 | 0 | 3 |
| 24024 | PART-003 | 2019-04-30 09:42:22 | 0 | 3 |
| 24023 | PART-004 | 2019-04-30 09:42:22 | 0 | 3 |
| 24022 | PART-005 | 2019-04-30 09:42:22 | 0 | 3 |
+-------+----------+---------------------+---------+---------+
Then, using PART-001 to make sure the flag_id is actually set:
MariaDB [mastern]> select `pro_flag_id` from `products` where `pro_part_number` = "PART-001";
+-------------+
| pro_flag_id |
+-------------+
| 3 |
+-------------+
So I'm not really sure what's going wrong, the logic and everything to me (MySQL newb) looks like it should work but it's giving me empty data. I also tried changing the INNER JOIN's to LEFT JOIN's but again, didn't work.
What am I doing wrong?
I managed to solve it. As I'm really reading the flag_history table, I changed the main SELECT focus to flag_history, revised SQL:
$sql = 'SELECT `old_val`, `new_val`, `ts`, `flags`.`name` AS flag_name, `pro_part_number`, `pro_title`
FROM `flag_history`
INNER JOIN `products` ON `pro_part_number` = `part`
INNER JOIN `flags` ON `pro_flag_id` = `flags`.`id`
WHERE `ts` <= :dateTs;';
I slowly added in the INNER JOIN's and managed to get it working when using flag_history. I think the issue actually may have been down to the id field. Being an ambiguous field (flags and flag_history have the id column). Weirdly, I wasn't getting that error though (maybe as I wasn't selecting). Either way managed to resolve it by being more specific with my select and going from the read-table rather than joining it.
On a further note, removing the flags segment from the JOIN shows the constraint error. So my guess is that it didn't show first time as I was reading from a different table.
I have a SQL table which looks like:
---------------------
| price |price_two |
---------------------
| 35.90 | 0 |
| 21.90 | 5 |
| 7.90 | 0 |
| 50.21 | 30.29 |
---------------------
And now I want to create a query in SQL which switches all values from price to price_two, where price_two is lower than price but NOT 0.
So, at the end I want this:
---------------------
| price |price_two |
---------------------
| 35.90 | 0 |
| 5 | 21.90 |
| 7.90 | 0 |
| 30.29 | 50.21 |
---------------------
Can I do that and if yes, how?
Greetings and Thank You!
Something like...
SQL Fiddle
UPDATE Price P
INNER JOIN Price P2
on P.Price = P2.Price
AND P2.Price_two = P.Price_two
AND P.Price_two < P.Price
AND P.Price_two != 0
SET P.Price = P2.Price_two,
P.Price_two = P2.Price;
We simply use a self join and use the 2nd table's data to update the first since we need to materialize the values and perform both updates at once or we lose the handle to original values!
I'm working on an application with has millions of data, and I need to list all the data in a page. At first I was fetching all the data in a single query and the pagination was done in the client side. But that takes almost 15 minutes to load the complete set. So I changed the code for fetching 10 rows per request and the pagination was done on server side. But still the performance is not up to the mark. So what all things should do to get the data quickly or what is the best way to handle huge data.
My query to fetch data :
UPDATED :
SELECT w.work_order_id,
(SELECT CONCAT(user_fname, ' ', user_lname) FROM users WHERE user_id = w.created_by) AS created_by,
CASE w.assignee WHEN 0 THEN 'All' WHEN -1 THEN 'Unassigned' ELSE CONCAT(u.user_fname, ' ', u.user_lname) END AS assignee
FROM FiveVan_work_orders w
LEFT JOIN users u ON (u.user_id = w.assignee)
WHERE ( w.work_order_status != 'Deleted' && w.work_order_status != 'Closed') ORDER BY w.created_on DESC LIMIT 0,10;
I have created index for the pages and this is the result of explaining the query
+----+--------------------+-------+--------+-------------------+---------------+---------+-------------------------------+--------+------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------+--------+-------------------+---------------+---------+-------------------------------+--------+------------------------------------------+
| 1 | PRIMARY | w | index | work_order_status | work_order_id | 790 | NULL | 340319 | Using where; Using index; Using filesort |
| 1 | PRIMARY | u | eq_ref | PRIMARY | PRIMARY | 4 | fivevan_loadtest.w.assignee | 1 | NULL |
| 2 | DEPENDENT SUBQUERY | users | eq_ref | PRIMARY | PRIMARY | 4 | fivevan_loadtest.w.created_by | 1 | NULL |
+----+--------------------+-------+--------+-------------------+---------------+---------+-------------------------------+--------+------------------------------------------+
WHERE ( w.work_order_status != 'Deleted' &&
w.work_order_status != 'Closed')
ORDER BY w.created_on DESC
If there is only one more value of work_order_status, then change that to
WHERE w.work_order_status = 'Open'
ORDER BY w.created_on DESC
and add
INDEX(work_order_status, created_on)
If there are multiple other values, the this may work (less well):
INDEX(created_on)
To get even better performance, you need to "remember where you left off" instead of using OFFSET. I discuss that in my pagination blog.
Why do this two queries return different result set when they have the same ORDER BY.
Only difference in query is that first time I user INNER JOIN an it takes about 5 seconds.
Second time I used LEFT JOIN and it took 0.05 seconds. In both cases they return exactly 43.000 rows, but tck.id order is different and I can't figure out why or in which way?
SELECT tck.*, acc.ac_name
FROM support_tickets tck
INNER JOIN support_ticket_accounts acc USING (id_support_ticket_account)
WHERE tck.id_company = 2 AND tck.st_status = 1 ORDER BY tck.st_priority DESC
Edit:
SELECT tck.*, acc.ac_name
FROM support_tickets tck
LEFT JOIN support_ticket_accounts acc ON tck.id_support_ticket_account = acc.id_support_ticket_account
WHERE tck.id_company = 2 AND tck.st_status = 1
ORDER BY tck.st_priority DESC;
+----+-------------+-------+--------+---------------+------------+---------+-------------------------------+-------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+------------+---------+-------------------------------+-------+-----------------------------+
| 1 | SIMPLE | tck | ref | id_company | id_company | 5 | const | 37586 | Using where; Using filesort |
| 1 | SIMPLE | acc | eq_ref | PRIMARY | PRIMARY | 4 | tck.id_support_ticket_account | 1 | |
+----+-------------+-------+--------+---------------+------------+---------+-------------------------------+-------+-----------------------------+
SELECT tck.*, acc.ac_name
FROM support_tickets tck
INNER JOIN support_ticket_accounts acc ON tck.id_support_ticket_account = acc.id_support_ticket_account
WHERE tck.id_company = 2 AND tck.st_status = 1
ORDER BY tck.st_priority DESC;
+----+-------------+-------+------+-------------------------------------+----------------------------+---------+-------------------------------+------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+--------------------------------------+---------------------------+---------+-------------------------------+------+---------------------------------+
| 1 | SIMPLE | acc | ALL | PRIMARY | NULL | NULL | NULL | 5 | Using temporary; Using filesort |
| 1 | SIMPLE | tck | ref | id_company,id_support_ticket_account | id_support_ticket_account | 5 | acc.id_support_ticket_account | 2085 | Using where |
+----+-------------+-------+------+--------------------------------------+---------------------------+---------+-------------------------------+------+---------------------------------+
I think using temporary is responsible for the delay (but I don't see why it's necessary for one query and not the other one). I think creating multi-column index should help:
CREATE INDEX filter
ON support_tickets(id_company, st_status, st_priority)
USING BTREE;
If you just ORDER BY tck.st_priority DESC multiple different recordsets are posible and can be returned, for each of both cases (left or inner). That is because you must have a lot of records that has the same st_priority so any of them can came in no particular order
Add more fields to the order by clause to give any record unique possible position and you will have same order on both querys.