I have a table which contains a standard auto-incrementing ID, a type identifier, a number, and some other irrelevant fields. When I insert a new object into this table, the number should auto-increment based on the type identifier.
Here is an example of how the output should look:
id type_id number
1 1 1
2 1 2
3 2 1
4 1 3
5 3 1
6 3 2
7 1 4
8 2 2
As you can see, every time I insert a new object, the number increments according to the type_id (i.e. if I insert an object with type_id of 1 and there are 5 objects matching this type_id already, the number on the new object should be 6).
I'm trying to find a performant way of doing this with huge concurrency. For example, there might be 300 inserts within the same second for the same type_id and they need to be handled sequentially.
Methods I've tried already:
PHP
This was a bad idea but I've added it for completeness. A request was made to get the MAX() number for the item type and then add the number + 1 as part of an insert. This is quick but doesn't work concurrently as there could be 200 inserts between the request for MAX() and that particular insert leading to multiple objects with the same number and type_id.
Locking
Manually locking and unlocking the table before and after each insert in order to maintain the increment. This caused performance issues due to the number of concurrent inserts and because the table is constantly read from throughout the app.
Transaction with Subquery
This is how I'm currently doing it but it still causes massive performance issues:
START TRANSACTION;
INSERT INTO objects (type_id,number) VALUES ($type_id, (SELECT COALESCE(MAX(number),0)+1 FROM objects WHERE type_id = $type_id FOR UPDATE));
COMMIT;
Another negative thing about this approach is that I need to do a follow up query in order to get the number that was added (i.e. searching for an object with the $type_id ordered by number desc so I can see the number that was created - this is done based on a $user_id so it works but adds an extra query which I'd like to avoid)
Triggers
I looked into using a trigger in order to dynamically add the number upon insert but this wasn't performant as I need to perform a query on the table I'm inserting into (which isn't allowed so has to be within a subquery causing performance issues).
Grouped Auto-Increment
I've had a look at grouped auto-increment (so that the number would auto-increment based on type_id) but then I lose my auto-increment ID.
Does anybody have any ideas on how I can make this performant at the level of concurrent inserts that I need? My table is currently InnoDB on MySQL 5.5
Appreciate any help!
Update: Just in case it is relevant, the objects table has several million objects in it. Some of the type_id can have around 500,000 objects assigned to them.
Use transaction and select ... for update. This will solve concurrency conflicts.
In Transaction with Subquery
Try to make index on column type_id
I think by making index on column type_id it will speed up your subquery.
DROP TABLE IF EXISTS my_table;
CREATE TABLE my_table
(id INT NOT NULL AUTO_INCREMENT PRIMARY KEY
,type_id INT NOT NULL
);
INSERT INTO my_table VALUES
(1,1),(2,1),(3,2),(4,1),(5,3),(6,3),(7,1),(8,2);
SELECT x.*
, COUNT(*) rank
FROM my_table x
JOIN my_table y
ON y.type_id = x.type_id
AND y.id <= x.id
GROUP
BY id
ORDER
BY type_id
, rank;
+----+---------+------+
| id | type_id | rank |
+----+---------+------+
| 1 | 1 | 1 |
| 2 | 1 | 2 |
| 4 | 1 | 3 |
| 7 | 1 | 4 |
| 3 | 2 | 1 |
| 8 | 2 | 2 |
| 5 | 3 | 1 |
| 6 | 3 | 2 |
+----+---------+------+
or, if performance is an issue, just do the same thing with a couple of #variables.
Perhaps an idea to create a (temporary) table for all rows with a common "type_id".
In that table you can use auto-incrementing for your num colomn.
Then your num shoud be fully trustable.
Then you can select your data and update your first table.
Related
I have a MariaDB table with an auto-incremented index, but also a "sortorder" field that controls the, well, sort order, when data is queried and displayed.
E.g.
id title sortorder
1 this 10
2 that 30
3 other 20
4 something 25
So far, so good. I'd like to create a function to automatically re-order these though - well, not re-order, but redo the values of the sortorder column per the existing order. The desired outcome from the above after running the function would be this:
id title sortorder
1 this 10
2 that 40
3 other 20
4 something 30
Is this something that can be done with an SQL statement in MariaDB (I have not found anything for that yet), or do I need to do it in my (php) application?
The logic for the new sort order values is based on the ordering by the sortorder column.
The reason for renumbering is that the sort order values are going to be manually maintained in the application, but it may be occasionally helpful to start with a clean slate. Users will be trained to "leave some room" in the values to allow for future edits.
On day one, "sortorder" will get (manually) populated with, say, 10, 20, 30, etc. Or possibly 100, 200, 300, etc. So that if they need to reorder things in the future, this will allow changing one item's sortorder value to say 25, to put it between the items with 20 and 30. Make sense?
But eventually, it's possible that the users could paint themselves into a corner, or at any rate make things confusing for themselves. It would be nice to build them a button that simply goes through the rows, and re-sets all the sortorder values, to preserve the existing row order but to make the values of sortorder be spaced evenly by intervals anew.
This would require some subqueries to be written inside. Steps of what I did are as follows:
Table name I used is tt. You need to change it according to your table name.
First is to get all rows in sorted order of sortorder column.
Second, declare a variable, say #serial_no and keep incrementing it by 1 on every selected row. This is an old school technique but I find it more readable.
Assign new sortorder values in this new parent select query. For now, I have just multiplied it's serial number(as in rank) by 10. You can adjust accordingly.
In your update query, inner join current copy of the table being updated with this select query and update the new sortorder column values correctly by matching them on id column.
Snippet:
update tt A
inner join (
select id, title, (#serial_no := #serial_no + 1) as serial_no,#serial_no * 10 as `sortorder`
from (
select *
from tt
order by sortorder asc
) temp_derived,(SELECT #serial_no := 0) as sn
) B
on A.id = B.id
set A.sortorder = B.sortorder
Update:
I just realised the control is completely shifted from user to DB. If you wish to update multiple rows with their new sortorder values, I wish to propose a workaround technique since I have never seen updating multiple rows with new values submitted from user in bulk(happy to learn if there exists one).
You need to map old values with new values, say in an associative array in PHP.
Start a DB transaction in MySQL.
Insert all new rows in bulk.
Delete all previous old rows in one go with IDs sent from PHP (in a prepared statement preferably with the previously mapped assoc array)
Commit the transaction.
Rollback ofcourse if something goes wrong.
Update: This solution works in MySQL 8.0, but not in MariaDB, because MariaDB's support for CTE doesn't support UPDATE statements. I'll leave this solution here for readers who use MySQL, but it doesn't work for MariaDB.
mysql> select * from NoOneEverNamesTheirTableInSqlQuestions;
+----+-----------+-----------+
| id | title | sortorder |
+----+-----------+-----------+
| 1 | this | 10 |
| 2 | that | 30 |
| 3 | other | 20 |
| 4 | something | 25 |
+----+-----------+-----------+
mysql> with cte as (
select id, row_number() over (order by sortorder) * 10 as new_sortorder
from NoOneEverNamesTheirTableInSqlQuestions
)
update NoOneEverNamesTheirTableInSqlQuestions join cte using (id)
set sortorder = new_sortorder;
Query OK, 2 rows affected (0.01 sec)
Rows matched: 4 Changed: 2 Warnings: 0
mysql> select * from NoOneEverNamesTheirTableInSqlQuestions;
+----+-----------+-----------+
| id | title | sortorder |
+----+-----------+-----------+
| 1 | this | 10 |
| 2 | that | 40 |
| 3 | other | 20 |
| 4 | something | 30 |
+----+-----------+-----------+
In MariaDB you can do:
update t
join (
select id, 10 * row_number() over (order by sortorder) as rn10
from t
) x on x.id = t.id
set t.sortorder = x.rn10;
Result:
id title sortorder
--- ---------- ---------
1 this 10
2 that 40
3 other 20
4 something 30
See running example at db<>fiddle.
I have a PHP script pulling a JSON file that is static and updates every 10 seconds. It has details about some events that happen and it just adds to the top of the JSON file. I then insert them into a MySQL database.
Because I have to pull every event every time I pull the file, I will only be inserting new events. The easy way would be to search for the event in the database (primary keys are not the same), but I am talking about ~4000 events every day, and I do not want that many queries just to see if it exists.
I am aware of INSERT IGNORE, but it looks like it only uses PRIMARY_KEY to do this.
What can I do (preferably easily) to prevent duplicates on two keys?
Example:
I have a table events with the following columns:
ID (irrelevant, really)
event_id (that I need to store from the source)
action_id (many action_ids belong to one event_id)
timestamp
whatever...
And my data is my JSON comes out on the first pull like this:
event_id|action_id|...
1 | 1
1 | 2
1 | 3
2 | 1
2 | 2
2 | 3
Then the next pull is this:
event_id|action_id|...
1 | 1
1 | 2
1 | 3
1** | 4**
1** | 5**
2 | 1
2 | 2
2 | 3
2** | 4**
I only want the rows marked with asterisks to be inserted, and the others to be ignored. Remember, primary_key column id is completely in this table, and I just use it for ubiquity.
What command can I use to "INSERT" every event I pull, but ONLY adding those that aren't duplicated by way of the two columns event_id and action_id.
Thanks.
Create a unique index of both columns.
CREATE
UNIQUE INDEX event_action
ON tablename (event_id, action_id)
Have a table that will be shared by multiple users. The basic table structure will be:
unique_id | user_id | users_index_id | data_1 | data_2 etc etc
With the id fields being type int and unique_id being an primary key with auto increment.
The data will be something like:
unique_id | user_id | users_index_id
1 | 234 | 1
2 | 234 | 2
3 | 234 | 3
4 | 234 | 4
5 | 732 | 1
6 | 732 | 2
7 | 234 | 5
8 | 732 | 3
How do I keep track of 'users_index_id' so that it 'auto increments' specifically for a user_id ?
Any help would be greatly appreciated. As I've searched for an answer but am not sure I'm using the correct terminology to find what I need.
The only way to do this consistently is by using a "before insert" and "before update" trigger. MySQL does not directly support this syntax. You could wrap all changes to the table in a stored procedure and put the logic there, or use very careful logic when doing an insert:
insert into table(user_id, users_index_id)
select user_id, count(*) + 1
from table
where user_id = param_user_id;
However, this won't keep things in order if you do delete or some updates.
You might find it more convenient to calculate the users_index_id when you query rather than in the database. You can do this using either subqueries (which are probably ok with the right indexes on the table) or using variables (which might be faster but can't be put into a view).
If you have an index on table(user_id, unique_id), then the following query should work pretty well:
select t.*,
(select count(*) from table t2 where t2.user_id = t.user_id and t2.unique_id <= t.unique_id
) as users_index_id
from table t;
You will need the index for non-abyssmal performance.
You need to find the MAX(users_index_id) and increment it by one. To avoid having to manually lock the table to ensure a unique key you will want to perform the SELECT within your INSERT statement. However, MySQL does not allow you to reference the target table when performing an INSERT or UPDATE statement unless it's wrapped in a subquery:
INSERT INTO users (user_id, users_index_id) VALUES (234, (SELECT IFNULL(id, 0) + 1 FROM (SELECT MAX(users_index_id) id FROM users WHERE user_id = 234) dt))
Query without subselect (thanks Gordon Linoff):
INSERT INTO users (user_id, users_index_id) SELECT 234, IFNULL((SELECT MAX(users_index_id) id FROM users WHERE user_id = 234), 0) + 1;
http://sqlfiddle.com/#!2/eaea9a/1/0
I have a table that records tickets that are separated by a column that denotes the "database". I have a unique key on the database and cid columns so that it increments each database uniquely (cid has the AUTO_INCREMENT attribute to accomplish this). I increment id manually since I cannot make two AUTO_INCREMENT columns (and I'd rather the AUTO_INCREMENT take care of the more complicated task of the uniqueness).
This makes my data look like this basically:
-----------------------------
| id | cid | database |
-----------------------------
| 1 | 1 | 1 |
| 2 | 1 | 2 |
| 3 | 2 | 2 |
-----------------------------
This works perfectly well.
I am trying to make a feature that will allow a ticket to be "moved" to another database; frequently a user may enter the ticket in the wrong database. Instead of having to close the ticket and completely create a new one (copy/pasting all the data over), I'd like to make it easier for the user of course.
I want to be able to change the database and cid fields uniquely without having to tamper with the id field. I want to do an UPDATE (or the like) since there are foreign key constraints on other tables the link to the id field; this is why I don't simply do a REPLACE or DELETE then INSERT, as I don't want it to delete all of the other table data and then have to recreate it (log entries, transactions, appointments, etc.).
How can I get the next unique AUTO_INCREMENT value (based on the new database value), then use that to update the desired row?
For example, in the above dataset, I want to change the first record to go to "database #2". Whatever query I make needs to make the data change to this:
-----------------------------
| id | cid | database |
-----------------------------
| 1 | 3 | 2 |
| 2 | 1 | 2 |
| 3 | 2 | 2 |
-----------------------------
I'm not sure if the AUTO_INCREMENT needs to be incremented, as my understanding is that the unique key makes it just calculate the next appropriate value on the fly.
I actually ended up making it work once I re-read an except on using AUTO_INCREMENT on multiple columns.
For MyISAM and BDB tables you can specify AUTO_INCREMENT on a
secondary column in a multiple-column index. In this case, the
generated value for the AUTO_INCREMENT column is calculated as
MAX(auto_increment_column) + 1 WHERE prefix=given-prefix. This is
useful when you want to put data into ordered groups.
This was the clue I needed. I simply mimic'd the query MySQL runs internally according to that quote, and joined it into my UPDATE query as such. Assume $new_database is the database to move to, and $id is the current ticket id.
UPDATE `tickets` AS t1,
(
SELECT MAX(cid) + 1 AS new_cid
FROM `tickets`
WHERE database = {$new_database}
) AS t2
SET t1.cid = t2.new_cid,
t1.database = {$new_database}
WHERE t1.id = {$id}
I have mysql table of some records, e.g.:
CREATE TABLE test (
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
value varchar NOT NULL
)
now, what I need is to generate unique sequence of 1, 2, ..., N in php script and store to another table... How to achieve this to be thread-safe and not creating doubles or skipping something?
I was wondering if some additional mysql table could be helpful, but I don't know how to create something like "separate autoincrements for each column value" or anything else...
test:
1 ... apples
2 ... oranges
3 ... lemons
some php script (accessed parallely by multiple users at the time):
save_next_fruit($_GET['fruit']);
will create some record in another tables with values like this:
saved_fruit:
ID | FRUIT(FK) | FRUIT_NO
1 1 1
2 1 2
3 2 1
4 3 1
5 3 2
6 1 3
7 3 3
8 2 2
9 1 4
10 2 3
11 1 5
12 2 4
13 1 6
14 3 4
15 3 5
other words, I need to do this (e.g. for fruit 3 (lemons)):
insert into saved_fruit (fruit, fruit_no) values (3, select MAX(fruit_no)+1 from saved_fruit where fruit = 3);
but in thread safe way (I understand that above command is not thread safe in MyISAM MySQL database)
Can you help?
Thanks
MyISAM does support this behavior. Create a two-column primary key, and make the second column auto-increment. It'll start over for each distinct value in the first column.
CREATE TABLE t (i INT, j INT AUTO_INCREMENT, PRIMARY KEY (i,j)) ENGINE=MyISAM;
INSERT INTO t (i) VALUES (1), (1), (2), (2), (1), (3);
SELECT * FROM t;
+---+---+
| i | j |
+---+---+
| 1 | 1 |
| 1 | 2 |
| 1 | 3 |
| 2 | 1 |
| 2 | 2 |
| 3 | 1 |
+---+---+
But if you think about it, this is only thread-safe in a storage engine that does table-level locking for INSERT statements. Because the INSERT has to search other rows in the table to find the max j value per the same i value. If other people are doing INSERTs concurrently, it creates a race condition.
Thus, the dependency on MyISAM, which does table-level locking on INSERT.
See this reference in the manual: http://dev.mysql.com/doc/refman/5.6/en/example-auto-increment.html under the section, MyISAM Notes.
There are a whole lot of good reasons not to use MyISAM. The deciding factor for me is MyISAM's tendency to corrupt data.
Re your comment:
InnoDB does not support the increment-per-group behavior described above. You can make a multi-column primary key, but the error you got is because InnoDB requires that the auto-increment column be the first column in a key of the table (it doesn't strictly have to be the primary key)
Regardless of the position of the auto-increment column in the multi-column key, it only increments when you use it with InnoDB; it does not number entries per distinct value in another column.
To do this with an InnoDB table, you'd have to lock the table explicitly for the duration of the INSERT, to avoid race conditions. You'd do your own SELECT query for the max value in the group you're inserting to. Then insert that value + 1.
Basically, you have to bypass the auto-increment feature and specify values instead of having them automatically generated.
As you using MyISAM you could to lock whole table.
LOCK TABLES `saved_fruit`;
-- Insert query with select.
UNLOCK TABLES;