I have a main table with a primary key (ÍD) and a linked one.
Existing Data
Main
ID | name
=========
1 | foo
2 | bar
3 | loo
4 | zoo
Linked
main_id
=======
1
1
2
2
There are connections to the first entries of main (1,2) in the linked table.
Now new data gets imported from the same structure:
Import Data
ID | name
=========
1 | new_foo
2 | new_bar
3 | new_loo
4 | new_zoo
and
main_id
=======
3
4
3
1
During the import process the IDs of the main table will get new ids (done by a script)
ID | name
=========
1 | foo
2 | bar
3 | loo
4 | zoo
5 | new_foo
6 | new_bar
7 | new_loo
8 | new_zoo
but the main_id will still have the ids from the imported data:
main_id
=======
1
1
2
2
3 => should be 7
4 => should be 8
3 => should be 7
1 => should be 5 => * comment below
*I cannot simple update linked like:
UPDATE linked SET main_id = 5 WHERE main_id = 1
as it would update the first two rows as well.
So how can I map these field with the new primary ID of main?
I could add a high number to the main_id before import like
UPDATE linked SET main_id = main_id + 10000000
do the import
apply the real ID
UPDATE linked SET main_id = %realID WHERE main_id = %importedID
return my temporary ID back to the original.
UPDATE linked SET main_id = main_id - 10000000 WHERE main_id > 10000000
The problem is pretty obvious: This doesn't work well (or at all) if the IDs are higher than 10.000.000 or the temporary id is higher than BIGINT (9223372036854775807).
It could work with a clone of the table but this may cause a problem in memory consumption as the linked table can get pretty big.
I'm sure there's a "best practice" way of doing this.
The steps are:
Pick last ID in Main table: select #lastId := max(ID) from Main;
Just shift all IDs, that are to be written in Linked table, by #lastId.
Something like this: insert into Linked select ID + #lastId from Imported_linked_data
No extra steps needed.
You can add the new entries in the Main table and let the AUTO_INCREMENT column generate the Id of the new row (as it should). For each row you cache the generated Id in an associative array, which maps the old Id to the new generated Id.
After that you insert the entries for the linked table. Before adding the new row you replace the old foreign key Id to the new generated id based on your created associative array.
Related
I have a field DB table structure like below:
Before the ID I have the PK_ID column, with this structure table is it possible to create a PHP scripts to generate something like:
PK_ID | ID | plan
1 | 1 | Plan Base
2 | 2 | Plan Base
3 | 1 | Plan medium
ID is autoincrement
What I want to do is, the ID value will start new sequence when new plan value added.
Thanks
You can set the counter:
ALTER TABLE tablename AUTO_INCREMENT = _NEW_ID_
https://dev.mysql.com/doc/refman/8.0/en/example-auto-increment.html
I would like to create a queue system that works in this way:
A user fills in a form where they will have to enter some data.
Click on Send and these data will be saved in a sql table.
Going to the index.php page will see a box containing a text like this: There are 4 requests in front of you, please wait a few minutes.
I have already tried to do such a thing, but going to create new requests the number "4" of the message grows.
This is because I created a query that counts all the results on the table.
$query = $mysql->query("SELECT COUNT(*) AS q FROM application_approve");
While I want it to count only the results above the request that sent the user.
id name text text2
1 First request dassasad dsadasas
2 Second request dassasad dsadasas
3 Third request dsadasdsas dsadasad
In the example above I would like to count only how many lines there are above the "Second Request": in this case 1.
Assuming your table has a PK (id) and references a user_id to identify which request belongs to which user and assuming there can only be a single request in the queue per user then your query would look something like the following.
SELECT COUNT(id) AS q FROM application_approve
WHERE id < (
SELECT id FROM application_approve
WHERE user_id = ?
)
This also assumes the PK id is an auto-incrementing key.
Given the user_id this query would return the number of rows above the given user's row (assuming they have one). Or, in other words, all ids less than the id of the given user.
For simplicity, let's assume this schema only has 2 columns (id and user_id):
mysql> SELECT * FROM application_approve;
+------+---------+
| id | user_id |
+------+---------+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
+------+---------+
3 rows in set (0.00 sec)
So in the given table, there are 3 users, each with 1 entry in the queue.
If we wanted to find which position user 2 is in the query would give us the following result:
mysql> SELECT COUNT(id) AS q FROM application_approve WHERE id < (SELECT id FROM application_approve WHERE user_id = 2);
+---+
| q |
+---+
| 1 |
+---+
1 row in set (0.00 sec)
I have a table with all my invoice items as packages:
Table: invoice_items
invoice_item_id | package_id | addon_1 | addon_2 | addon_3 | ...
----------------|------------|---------|---------|
1 | 6 | 2 | 5 | 3 |
Then my other table:
Table: addons
addon_id | addon_name | addon_desc |
----------|--------------|--------------------------|
1 | Dance Lights | Brighten up the party... |
2 | Fog Machine | Add some fog for an e... |
Instead of taking up space storing the addon name in my invoice_items table, I'd like to just include the addon_id in the addon_1, addon_2, etc columns.
How do I then get the name of the addon when doing a query for invoice_item rows?
Right now I just have it programmed into the page that if addon_id == 1, echo "Dance Lights", etc but I'd like to do it in the query. Here is my current query:
$invoice_items_SQL = "
SELECT invoice_items.*, packages.*
FROM `invoice_items`
INNER JOIN packages ON invoice_items.invoice_item_id = packages.package_id
WHERE `event_id` = \"$event_id\"
";
So I'm able to do this with packages, but only because there's just one package_id per row, but there are up to 9 addons :(
The most direct way of doing it is to join onto the table multiple times. That's a bit naff though because you'll write almost the same thing 9 times.
Another, better way would be to restructure your tables - you need another table with 2 data columns: invoice_id and addon_id. You then need either an auto-inc primary column, or use both of those existing columns as a dual primary key. So this is a many-to-many junction table.
From there you can can query without having 9 repetitive joins, but you will get a row of each package for every addon it has (so if it has three addons it will appear three times in the results). And then from there you can use GROUP_CONCAT to concatenate the names of the addons into a single field so that you only get one row per invoice.
I need to store and retrieve items of a course plan in sequence. I also need to be able to add or remove items at any point.
The data looks like this:
-- chapter 1
--- section 1
----- lesson a
----- lesson b
----- drill b
...
I need to be able to identify the sequence so that when the student completes lesson a, I know that he needs to move to lesson b. I also need to be able to insert items in the sequence, like say drill a, and of course now the student goes from lesson a to drill a instead of going to lesson b.
I understand relational databases are not intended for sequences. Originally, I thought about using a simple autoincrement column and use that to handle the sequence, but the insert requirement makes it unworkable.
I have seen this question and the first answer is interesting:
items table
item_id | item
1 | section 1
2 | lesson a
3 | lesson b
4 | drill a
sequence table
item_id | sequence
1 | 1
2 | 2
3 | 4
4 | 3
That way, I would keep adding items in the items table with whatever id and work out the sequence in the sequence table. The only problem with that system is that I need to change the sequence numbers for all items in the sequence table after an insertion. For instance, if I want to insert quiz a before drill a I need to update the sequence numbers.
Not a huge deal but the solutions seems a little overcomplicated. Is there an easier, smarter way to handle this?
Just relate records to the parent and use a sequence flag. You will still need to update all the records when you insert in the middle but I can't really think of a simple way around that without leaving yourself space to begin with.
items table:
id | name | parent_id | sequence
--------------------------------------
1 | chapter 1 | null | 1
2 | section 1 | 1 | 2
3 | lesson a | 2 | 3
4 | lesson b | 2 | 5
5 | drill a | 2 | 4
When you need to insert a record in the middle a query like this will work:
UPDATE items SET sequence=sequence+1 WHERE sequence > 3;
insert into items (name, parent_id, sequence) values('quiz a', 2, 4);
To select the data in order your query will look like:
select * from items order by sequence;
I have a table which contains a standard auto-incrementing ID, a type identifier, a number, and some other irrelevant fields. When I insert a new object into this table, the number should auto-increment based on the type identifier.
Here is an example of how the output should look:
id type_id number
1 1 1
2 1 2
3 2 1
4 1 3
5 3 1
6 3 2
7 1 4
8 2 2
As you can see, every time I insert a new object, the number increments according to the type_id (i.e. if I insert an object with type_id of 1 and there are 5 objects matching this type_id already, the number on the new object should be 6).
I'm trying to find a performant way of doing this with huge concurrency. For example, there might be 300 inserts within the same second for the same type_id and they need to be handled sequentially.
Methods I've tried already:
PHP
This was a bad idea but I've added it for completeness. A request was made to get the MAX() number for the item type and then add the number + 1 as part of an insert. This is quick but doesn't work concurrently as there could be 200 inserts between the request for MAX() and that particular insert leading to multiple objects with the same number and type_id.
Locking
Manually locking and unlocking the table before and after each insert in order to maintain the increment. This caused performance issues due to the number of concurrent inserts and because the table is constantly read from throughout the app.
Transaction with Subquery
This is how I'm currently doing it but it still causes massive performance issues:
START TRANSACTION;
INSERT INTO objects (type_id,number) VALUES ($type_id, (SELECT COALESCE(MAX(number),0)+1 FROM objects WHERE type_id = $type_id FOR UPDATE));
COMMIT;
Another negative thing about this approach is that I need to do a follow up query in order to get the number that was added (i.e. searching for an object with the $type_id ordered by number desc so I can see the number that was created - this is done based on a $user_id so it works but adds an extra query which I'd like to avoid)
Triggers
I looked into using a trigger in order to dynamically add the number upon insert but this wasn't performant as I need to perform a query on the table I'm inserting into (which isn't allowed so has to be within a subquery causing performance issues).
Grouped Auto-Increment
I've had a look at grouped auto-increment (so that the number would auto-increment based on type_id) but then I lose my auto-increment ID.
Does anybody have any ideas on how I can make this performant at the level of concurrent inserts that I need? My table is currently InnoDB on MySQL 5.5
Appreciate any help!
Update: Just in case it is relevant, the objects table has several million objects in it. Some of the type_id can have around 500,000 objects assigned to them.
Use transaction and select ... for update. This will solve concurrency conflicts.
In Transaction with Subquery
Try to make index on column type_id
I think by making index on column type_id it will speed up your subquery.
DROP TABLE IF EXISTS my_table;
CREATE TABLE my_table
(id INT NOT NULL AUTO_INCREMENT PRIMARY KEY
,type_id INT NOT NULL
);
INSERT INTO my_table VALUES
(1,1),(2,1),(3,2),(4,1),(5,3),(6,3),(7,1),(8,2);
SELECT x.*
, COUNT(*) rank
FROM my_table x
JOIN my_table y
ON y.type_id = x.type_id
AND y.id <= x.id
GROUP
BY id
ORDER
BY type_id
, rank;
+----+---------+------+
| id | type_id | rank |
+----+---------+------+
| 1 | 1 | 1 |
| 2 | 1 | 2 |
| 4 | 1 | 3 |
| 7 | 1 | 4 |
| 3 | 2 | 1 |
| 8 | 2 | 2 |
| 5 | 3 | 1 |
| 6 | 3 | 2 |
+----+---------+------+
or, if performance is an issue, just do the same thing with a couple of #variables.
Perhaps an idea to create a (temporary) table for all rows with a common "type_id".
In that table you can use auto-incrementing for your num colomn.
Then your num shoud be fully trustable.
Then you can select your data and update your first table.