Just looking for some tips and pointers for a small project I am doing. I have some ideas but I am not sure if they are the best practice. I am using mysql and php.
I have a table called nomsing in the database.
It has a primary key called row id which is an integer.
Then I have about 8 other tables referencing this table.
That are called nomplu, accsing,accplu, datsing, datplu for instance.
Each has a column that references the primary key of nomsing.
Withing my php code I have all the information to insert into the tables except one thing , the row id primary key of the nomsing table. So that php generates a series of inserts like the following.
INSERT INTO nomsing(word,postress,gender) VALUES (''велосипед","8","mask").
INSERT INTO nomplu(word,postress,NOMSING?REFERENCE) VALUES (''велосипеды","2",#the reference to the id of the first insert#).
There are more inserts but this one gets the point across. The second insert should reference the auto generated id for the first insert. I was this to work as a transaction so all inserts should complete or none.
One idea I have is to not auto generate the id and generate it myself in php. That way would know the id given before the transaction but then I would have to check if the id was already in the db.
Another idea I have is to do the first insert and then query for the row id of that insert in php and then make the second insert. I mean both should work but they don't seem like an optimal solution. I am not too familiar with the database transactional features but what would be the best approach to do in this case. I don't like the idea of inserting then querying for the id and then running the rest of the queries. Just seems very inefficient or perhaps I am wrong.
Just insert a row in the master table. Then you can fetch the insert id ( lastInserId when on PDO) and use that to populate your other queries.
You could use the php version as given by JvdBerg , or Mysql's LAST_INSERT_ID. I usually use the former option.
See a similar SO question here.
You could add a new column to the nomsing table, called 'insert_order' (or similar) with a default value of 0, then instead of generating one SQL statement per insert create a bulk insert statement e.g.
INSERT INTO nomsing(word,postress,gender, insert_order)
VALUES (''велосипед","8","mask",1), (''abcd'',"9","hat",2).....
you generate the insert_order number with a counter in your loop starting at one. Then you can perform one SELECT on the table to get the ids e.g.
SELECT row_id
FROM nomsing
WHERE insert_order > 0;
now you have all the IDs you can now do a bulk insert for your following queries. At the end of your script just do an update to reset the insert_order column back to 0
UPDATE nomsing SET insert_order = 0 WHERE insert_order > 0;
It may seem messy to add an extra column to do this but it will add a significant speed increase over performing one query at a time.
Related
I have a table called INVOICES that receives entries from a PHP script. It has many columns, but the two most relevant are INVOICE_ID and INVOICE_TYPE. Basically the INVOICE_TYPE is a number from 0 to 3, which designates different types of invoices.
Up to this point, everything ran smoothly until two users submitted invoices while the server had a hiccup and wrote both in as the same INVOICE_ID. The reason for this is the PHP script reads the MAX INVOICE_ID of the INVOICE_TYPE, then adds 1, then inserts the new row with that INVOICE_ID. In essence, it is programmatically a primary key. 99.9% of the time it worked, but that one time it was a problem.
I have tried finding SQL solutions but do not have sufficient knowledge of it. I have tried doing it myself in an SQL query to read the MAX, increment, and the insert but just throws an exception that you cannot select and insert from the same table at once.
What I'm wondering is if there is an auto-increment that could be conditional to the INVOICE_TYPE, to only increment if the type is matched. Any suggestions would help at this point.
An unique index over the two columns (INVOICE_ID, INVOICE_TYPE) will make one of such hiccupy queries fail.
CREATE UNIQUE INDEX id_type_unique ON INVOICES (INVOICE_ID, INVOICE_TYPE);
INSERT INTO INVOICES (INVOICE_ID, INVOICE_TYPE) VALUES (1, 5); -- okay
INSERT INTO INVOICES (INVOICE_ID, INVOICE_TYPE) VALUES (1, 5); -- error
If you insert only one row to one table at once simplest solution is to apply unique index on both columns.
CREATE UNIQUE INDEX invoice_id_type_unique
ON INVOICES(INVOICE_ID,INVOICE_TYPE);
But if you execute more queries based on the same data you need to use transactions to prevent modifying/inserting only part of data.
START TRANSACTION;
SELECT #invoice_id:=MAX(INVOICE_ID) FROM INVOICES WHERE INVOICE_TYPE=1;
INSERT INTO INVOICES (...,#invoice_id,...);
...
... #OTHER QUERIES UPDATING DATA
COMMIT;
make (INVOICE_ID) as UNIQUE, this will solved your problem, sql will not allowed duplicate value in same column.
The way i like it is to create a table to handle all the sequences and a stored procedure that i can call with the name of the sequence that i like to know the next value, something similar to:
START TRANSACTION;
SELECT value INTO result FROM Sequences WHERE name like paramSeqName FOR UPDATE;
UPDATE Sequences SET value = value + 1 WHERE name like paramSeqName;
COMMIT;
There is a good example here:
http://www.microshell.com/database/mysql/emulating-nextval-function-to-get-sequence-in-mysql/
working on the PHP project related to web scraping and my aim is to store the data into the mysql database,i'm using unique key index on 3 indexes in 9 columns table and records are more than 5k.
should i check for unique data at program level like putting values in arrays and then comparing before inserting into database ?
is there any way so that i can speed up my database insertion ?
Never ever create a duplicate table this is a anti SQL pattern and it makes it more difficult to work with your data.
Maybe PDO and prepared statement will give you a little boost but dont expect wonders from it.
multible INSERT IGNORE may also give you a little boost but dont expect wonders from it.
You should generate a multiinsert query like so
INSERT INTO database.table (columns) VALUES (values),(values),(values)
Keep in mind to keep under the max packet size that mysql will have.
this way the index file have to be updated once.
You could create a duplicate of the table that you currently have except no indices on any field. Store the data in this table.
Then use events to move the data from the temp table into the main table. Once the data is moved to the main table then delete if from the temp table.
you can follow your updates with triger. You should do update table and you have to right trigger for this table.
use PDO, mysqli_* function, to increase insertion into database
You could use "INSERT IGNORE" in your query. That way the record will not be inserted if any unique constraints are violated.
Example:
INSERT IGNORE INTO table_name SET name = 'foo', value = 'bar', id = 12345;
I am setting up an uploader (using php) for my client where they can select a CSV (in a pre-determined format) on their machine to upload. The CSV will likely have 4000-5000 rows. Php will process the file by reading each line of the CSV and inserting it directly into the DB table. That part is easy.
However, ideally before appending this data to the database table, I'd like to review 3 of the columns (A, B, and C) and check to see if I already have a matching combo of those 3 fields in the table AND IF SO I would rather UPDATE that row rather than appending. If I DO NOT have a matching combo of those 3 columns I want to go ahead and INSERT the row, appending the data to the table.
My first thought is that I could make columns A, B, and C a unique index in my table and then just INSERT every row, detect a 'failed' INSERT (due to the restriction of my unique index) somehow and then make the update. Seems that this method could be more efficient than having to make a separate SELECT query for each row just to see if I have a matching combo already in my table.
A third approach may be to simply append EVERYTHING, using no MySQL unique index and then only grab the latest unique combo when the client later queries that table. However I am trying to avoid having a ton of useless data in that table.
Thoughts on best practices or clever approaches?
If you make the 3 columns the unique id, you can do an INSERT with ON DUPLICATE KEY.
INSERT INTO table (a,b,c,d,e,f) VALUES (1,2,3,5,6,7)
ON DUPLICATE KEY UPDATE d=5,e=6,f=7;
You can read more about this handy technique here in the MySQL manual.
If you add a unique index on the ( A, B, C ) columns, then you can use REPLACE to do this in one statement:
REPLACE works exactly like INSERT,
except that if an old row in the table
has the same value as a new row for a
PRIMARY KEY or a UNIQUE index, the old
row is deleted before the new row is
inserted...
I have to insert data into two tables, Items and Class_Items. (A third table, Classes is related here, but is not being inserted into).
The primary key of Items is Item_ID, and it's an auto-incrementing integer. Aside from this primary key, there are no unique fields in Items. I need to know what the Item_ID is to match it to Classes in Class_Items.
This is all being done through a PHP interface. I'm wondering what the best way is to insert Items, and then match their Item_ID's into Class_Items. Here are the two main options I see:
INSERT each Item, then use mysql_insert_id() to get its Item_ID for the Class_Items INSERT query. This means one query for every Item (thousands of queries in total).
Get the next Autoincrement ID, then LOCK the Class_Items table so that I can just keep adding to an $item_id variable. This would mean just two queries (one for the Items, one for the Class_Items)
Which way is best and why? Also, if you have an unlisted alternative I'm open to whatever is most efficient.
The most efficient is probably going to be to use parameterized queries. That would require using the mysqli functions, but if you're to the point of needing to optimize this kind of query you should think about being there anyway.
No matter how you cut it, you've got two inserts to make. Doing the first, grabbing the new ID value as you've described (which imposes insignificant overhead, because the value is on hand to mysql already,) and using it in the second insert is pretty minimal.
I would investigate using stored procedures and/or transactions to make sure nothing bad happens.
I'm working on a project with mysql and what I did is the following (without using autoincrement fields):
1- I created a table called SEQUENCE with one field of type BIGINT called VALUE with an initial value of 1. This table will store the id value that will be incremented each time you insert a new record.
2- Create a store procedure and handle the id increment inside it within a transaction.
Here is an example.
CREATE PROCEDURE `SP_registerUser`(
IN _username VARCHAR(40),
IN _password VARCHAR(40),
)
BEGIN
DECLARE seq_user BIGINT;
START TRANSACTION;
#Validate that user does not exist etc..........
#Register the user
SELECT value FROM SEQUENCE INTO seq_user;
UPDATE SECUENCE SET value = value + 1;
INSERT INTO users VALUES(seq_user, _username, SHA1(_password));
INSERT INTO user_info VALUES(seq_user, UTC_TIMESTAMP());
COMMIT;
END //
In my case I want to store the user id in two different tables (users and user_info)
I'm trying to keep the database tables for a project I'm working on nice and normalized, but I've run into a problem. I'm trying to figure out how I can insert a row into a table and then find out what the value of the auto_incremented id column was set to so that I can insert additional data into another table. I know there are functions such as mysql_insert_id which "get the ID generated from the previous INSERT operation". However, if I'm not mistaken mysql_insert_id just returns the ID of the very last operation. So, if the site has enough traffic this wouldn't necessarily return the ID of the query you want since another query could have been run between when you inserted the row and look for the ID. Is this understanding of mysql_insert_id correct? Any suggestions on how to do this are greatly appreciated. Thanks.
LAST_INSERT_ID() has session scope.
It will return the identity value inserted in the current session.
If you don't insert any rows between INSERT and LAST_INSERT_ID, then it will work all right.
Note though that for multiple value inserts, it will return the identity of the first row inserted, not the last one:
INSERT
INTO mytable (identity_column)
VALUES (NULL)
SELECT LAST_INSERT_ID()
--
1
INSERT
INTO mytable (identity_column)
VALUES (NULL), (NULL)
/* This inserts rows 2 and 3 */
SELECT LAST_INSERT_ID()
--
2
/* But this returns 2, not 3 */
You could:
A. Assume that won't be a problem and use mysql_insert_id
or
B. Include a timestamp in the row and retrieve the last inserted ID before inserting into another table.
The general solution to this is to do one of two things:
Create a procedural query that does the insert and then retrieves the last inserted id (using, ie. LAST_INSERT_ID()) and returns it as output from the query.
Do the insert, do another insert where the id value is something like (select myid from table where somecolumnval='val')
2b. Or make the select explicit and standalone, and then do the other inserts using that value.
The disadvantage to the first is that you have to write a proc for each of these cases. The disadvantage to the second is that some db engines don't accept that, and it clutters your code, and can be slow if you have to do a where on multiple columns.
This assumes that there may be inserts between your calls that you have no control over. If you have explicit control, one of the other solutions above is probably better.