I'm running a mid-size system (PHP + MySQL) and I have a table stores all user's friends (this information comes from Facebook). The table now have about 1 million lines.
Every time user logs in, the system DELETEs all user`s friends from this table, and then insert it again using data downloaded from Facebook, so it's coded like this:
1) Run Facebook Query and store results in an object.
2) If query wasn`t successful, end program.
3) DELETE FROM UserFriends WHERE idUser = $idUser
4) INSERT INTO UserFriends (idUser, FriendFacebookId) VALUES
($idUser, $objFB[x]),
($idUser, $objFB[x2]),
($idUser, $objFB[x3])
...
* It generates the query inside a loop, so it INSERT all lines in one query only)
Sometimes (1 in ~2000) running the INSERT statement returns the following error:
1062 : Duplicate Entry for '2030-0202001910' for key 'PRIMARY'
The '2030-0202001910' is the User's ID and the FriendFacebookID... it's always the first set of data in the INSERT statement.
So the question is: I've just deleted everything for this user, the table is supposed to be empty for this user, but then when I try to insert data for this user, it returns me this message. Why this happens and how can I avoid it?
Please note: This program runs hundreds times per hour, and the error occurs only few times... If the same user try to log in again, the program works like expected... So i'm thinking that can be some "delay" between the DELETE command and the data been removed from table... I don't know.
Useful information:Table is InnoDB and I'm not using transactions.
Try this instead of insert: INSERT IGNORE
It is meant to mean 'ignore duplicates' so it never creates the possibility of creating a collision.
See if that works for you.
You could also use INSERT...ON DUPLICATE KEY UPDATE
...or simply: REPLACE
instead of the plain INSERT command
Related
We've got a web application that takes a uniquely generated workshop ID and calls a procedure (using php) and this procedure inserts it into a sql server table. This table has a clustered index on column workshopID and is set to unique.
This morning we had a user report that he got the following error code on his page:
Cannot insert duplicate key row in object 'dbo.table with unique
index 'ClusteredIndex-Wrkshp'. The duplicate key value is (Z9C1Am)
Obviously this suggests that I'm trying to insert Z9C1Am into the table and that value already exists---HOWEVER, when we do a simple lookup on that value, this value did not exist so I took the same stored procedure that the website code calls and used it to inserted Z9C1Am (using SSMS) into the table without any problem.
I can also get onto this application without any problems with this error; however, this same user called again and said he had the same issue (on the same computer) in another session. This time it had a different wordshopID in the error, but once again, this did not exist in the database.
I don't believe this has anything to do with inserting a duplicate key, rather, it is a phantom error.
Any suggestions on how to confirm this and how to track the actual error?
My gut reaction is this must be a browser related problem; however, all the code that interacts with the sql database is server side so this theory doesn't make much sense.
Thanks for the responses guys!
Greg's comment made me start looking harder at my dependencies and I have a join statement inside of my insert statement. The table I was joining to was supposed to contain unique values and approx. 25,000 of them are; however I had one set of duplicates causing the error.
I need to update and/or insert in a Postgres DB, a String value with letters and numbers, which must be incremented of +1 (through php strings functions) everytime there is an UPDATE.
I need to LOCK this table, in order for the php to complete its flow in inserting or updating it avoiding others that open the page to receive the same result of the SELECT.
The second arrived, would wait for the first to finish.
The second arrived, will ALWAYS make an UPDATE.
This update could generate a FILENAME0023, if there were 22 updates after the first insertion.
There could be more connections at the same time, I need to reserve the first result of the SELECT for the first one who connected to this php page.
The flow would be:
LOCK
SELECT column FROM table WHERE column = 'FILENAME0001';
IF NOT EXIST { INSERT INTO table (column) VALUES ($my_column); }
ELSE { UPDATE table SET column = '$my_new_column' WHERE column = '$my_column'; }
UNLOCK
The variable $my_new_column is a SUBSTR php function that would cut the number part of the string and will then be incremented of +1.
This link is helping me a lot: THIS.
But it does not contain everything.
I also tried working with stored procedure: LINK
But I should work on the php when it is an update because I cannot increment a DB value like shown HERE, because I do not have a INT but a string and I can not change this
Anyone who can help me?
I'd like to share my code, but believe me I'd rather start fresh, all the codes I tried are leading to nowhere.
you should try to use
`INSERT ON ON DUPLICATE KEY UPDATE`
Here link for more example about it
I'm trying my make an invitation system together with the Facebook PHP APK. What I basically want is when you have invited your friends, Facebook redirects back to your website with an array containing userIDs from friends that the user invited. I wish to loop through them and insert the invitations into a database. The looping and all is already under control but the database query isn't.
This is what my current query looks like:
SELECT `iID` FROM `invitations` WHERE `iSender` = 'SENDER' AND `iReceiver` = 'RECEIVER';
If that returns zero rows I process this query:
INSERT INTO `invitations` (`iSender`, `iReceiver`) VALUES ('SENDER', 'RECEIVER');
And then I check if they're already signed up to my website:
SELECT `uID` FROM `users` WHERE `uOAuthID` = 'RECEIVER';
If it returns more then 1 row I run the following and final query:
UPDATE `invitations` SET `iProcessed` = 1 WHERE `iReceiver` = 'RECEIVER';
So basically this is how the process is currently shaped:
If the user hasn't already has been invited by the inviter we insert the invitation into the database.
Then we check if the invited user already is signed up.
If he is signed up we update the invitation and say that it already has been processed.
I guess there's a better and faster method to do this with just maybe 1 or 2 queries. I've tried using INSERT IGNORE and ON DUPLICATE but that just gave me errors and gave up.
I hope that you understand what I'm looking for. Thank you all for your time!
There are to thing that suite your needs:
INSERT ... ON DUPLICATE KEY UPDATE Syntax:
If you specify ON DUPLICATE KEY UPDATE, and a row is inserted that
would cause a duplicate value in a UNIQUE index or PRIMARY KEY, an
UPDATE of the old row is performed. For example, if column a is
declared as UNIQUE and contains the value 1, the following two
statements have identical effect:
INSERT INTO table (a,b,c) VALUES (1,2,3) ON DUPLICATE KEY UPDATE
c=c+1;
INSERT IGNORE:
If you use the IGNORE keyword, errors that occur while executing the
INSERT statement are ignored. For example, without IGNORE, a row that
duplicates an existing UNIQUE index or PRIMARY KEY value in the table
causes a duplicate-key error and the statement is aborted. With
IGNORE, the row still is not inserted, but no error occurs.
This is my db structure:
ID NAME SOMEVAL API_ID
1 TEST 123456 A123
2 TEST2 223232 A123
3 TEST3 918922 A999
4 TEST4 118922 A999
I'm filling it using a function that calls an API and gets some data from an external service.
The first run, I want to insert all the data I get back from the API. After that, each time I run the function, I just want to update the current rows and add rows in case I got them from the API call and are not in the db.
So my initial thought regarding the update process is to go through each row I get from the API and SELECT to see if it already exists.
I'm just wondering if this is the most efficient way to do it, or maybe it's better to DELETE the relevant rows from the db and just re-inserting them all.
NOTE: each batch of rows I get from the API has an API_ID, so when I say delete the rows i mean something like DELETE FROM table WHERE API_ID = 'A999' for example.
If you retrieving all the rows from the service i recommend you the drop all indexes, truncate the table, then insert all the data and recreate indexes.
If you retrieving some data from the service i would drop all indexes, remove all relevant rows, insert all rows then recreate all indexes.
In such scenarios I'm usually going with:
start transaction
get row from external source
select local store to check if it's there
if it's there: update its values, remember local row id in list
if it's not there: insert it, remember local row id in list
at the end delete all rows that are not in remembered list of local row ids (NOT IN clause if the count of ids allows for this, or other ways if it's possible that there will be many deleted rows)
commit transaction
Why? Because usually I have local rows referenced by other tables, and deleting them all would break the references (not to mention deletete cascade).
I don't see any problem in performing SELECT, then deciding between an INSERT or UPDATE. However, MySQL has the ability to perform so-called "upserts", where it will insert a row if it does not exist, or update an existing row otherwise.
This SO answer shows how to do that.
I would recommend using INSERT...ON DUPLICATE KEY UPDATE.
If you use INSERT IGNORE, then the row won't actually be inserted if it results in a duplicate key on API_ID.
Add unique key index on API_ID column.
If you have all of the data returned from the API that you need to completely reconstruct the rows after you delete them, then go ahead and delete them, and insert afterwards.
Be sure, though, that you do this in a transaction, and that you are using an engine that supports transactions properly, such as InnoDB, so that other clients of the database don't see rows missing from the table just because they are going to be updated.
For efficiency, you should insert as many rows as you can in a single query. Much faster that way.
BEGIN;
DELETE FROM table WHERE API_ID = 'A987';
INSERT INTO table (NAME, SOMEVAL, API_ID) VALUES
('TEST5', 12345, 'A987'),
('TEST6', 23456, 'A987'),
('TEST7', 34567, 'A987'),
...
('TEST123', 123321, 'A987');
COMMIT;
I have a form and a database table named reports. I have a date field (primary key) and a textarea named changes to say what's been changed on that date. If the date is the same I want to be able to overwrite the information in the 'changes' column for that date.
My insert command, which works on its own, is as follows:
mysql_query("
INSERT INTO reports (thedate,changes)
VALUES ('$_POST[thedate]','$_POST[changes]')
");
I understand that I'll need to use ON DUPLICATE KEY UPDATE after my INSERT but after numerous attempts I cannot get it right. Not only do things no update but it seems to break my insert command so even a new row isn't added to the database.
Apologies if this is a duplicate question. After lots of searching and lots of trying I cannot get it to work.
Have a look att REPLACE.
REPLACE works exactly like INSERT, except that if an old row in the
table has the same value as a new row for a PRIMARY KEY or a UNIQUE
index, the old row is deleted before the new row is inserted.
Note that if you have a foreign key with an action ON DELETE it will be triggered when using REPLACE since it does a delete followed by an insert .
Using ON DUPLICATE KEY it could look like this:
INSERT reports (thedate, changes) VALUES ('$_POST[thedate]', '$_POST[changes]')
ON DUPLICATE KEY UPDATE changes = '$_POST[changes]'
This is the plain SQL query:
INSERT INTO reports (thedate, changes)
VALUES ('2011-11-10', 'Lorem ipsum')
ON DUPLICATE KEY UPDATE changes=VALUES(changes)
http://dev.mysql.com/doc/refman/5.5/en/insert-on-duplicate.html
Now, you absolutely need to sit down and try to understand what SQL is and how it interacts with PHP and differs from it. You are using PHP to compose strings that happen to be code from another language called SQL. The way you are doing it, the resulting code can be valid SQL or not, and it'll depend of the arbitrary data sent by any anonymous visitor. In the best case, your script will crash. In the worse case, the visitor will be able to read confidential data or alter your database. Here's the example in the manual page for mysql_query():
// This could be supplied by a user, for example
$firstname = 'fred';
$lastname = 'fox';
// Formulate Query
// This is the best way to perform an SQL query
// For more examples, see mysql_real_escape_string()
$query = sprintf("SELECT firstname, lastname, address, age FROM friends
WHERE firstname='%s' AND lastname='%s'",
mysql_real_escape_string($firstname),
mysql_real_escape_string($lastname));
// Perform Query
$result = mysql_query($query);
This should work:
mysql_query("INSERT INTO reports (thedate,changes) VALUES ('$_POST[thedate]','$_POST[changes]') ON DUPLICATE KEY UPDATE changes=VALUES(changes)");
Disadvantage of using REPLACE is that it's not standard SQL but a MySQL extension. Beside that, when using auto incremented columns, REPLACE will reinsert with a different value. I won't recommend it to anyone.
Edit: ON DUPLICATE KEY isn't standard SQL either. Sorry!