I have application that receives multiple requests from external sources (invoices from point-of-sale units). It gets tens of requests per second, and some of those requests are the same (have same request body).
request data is transformed and saved to two associated tables (foreign key). if record already exists (queried by unique composite key), record is updated, otherwise record is added.
the problem is that sometimes if two requests with same body are received at the same time app throws exception that unique key already exists and it can't insert it:
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry
It's probably some kind of race condition in MySQL but can't figure it out
Whole process is wrapped in Laravel transaction. I tried setting different isolation levels. SERIALIZABLE resolves that issue but then I get lot of deadlock exceptions, and as I can see sometimes record is not saved at all.
This is simple, what happen here, you have some value declared as "UNIQUE" or "Primary Key" or something like that, and you are trying to insert again, some key restriction it's blocking the insert, this should be good, you avoid duplicate entries in your database, but what you need to do its check if the data what you are trying to insert exist before in the database, not all the columns, you should ask for your keys or key combination, I can not help you more if I don't know the data or the table...
Related
I have a table in which the primary key is a 20 character VARCHAR field that gets generated in PHP before getting inserted into the table. The key generation logic uses grouping and sequencing mechanism as given below.
SELECT
SUBSTR(prod_code, 15) AS prod_num
FROM
items
, products
WHERE
items.cat_type = $category
AND items.sub_grp = $sub_grp
AND items.prod_id = products.prod_id
ORDER BY
prod_num DESC LIMIT 1
The prod_num thus got is incremented in PHP and prefixed with a product code to create a unique primary key. However multiple users can do the transaction concurrently for same category and sub_group leading to same keys generated for those users. This may lead to duplicate key error, as its a unique primary key. What is the best way to handle such a situation?
Don't use "Smart IDs".
Smart IDs were all the rage in the 1980s, and went out of fashion for several reasons:
The only requirement of a PK is that is has to be unique. A PK doesn't need to have a format, or to be sexy or good looking. Their specific sequence, case, or composition is not relevant and actually counter-productive.
They are not relational. Parts of the ID could establish a relationship with other tables and that can cause a lot of issues. This goes against Normal Forms defined in database design.
Now, if you still need a Smart ID, then create a secondary column (that can also be unique) and then populate it after the row is created. If you are facing thread safety issues, you can run a single deferred process that will assign nice looking values after a few minutes. Alternatively, you can implement a queue, that can resolve this is seconds.
Agree with "The Impaler".
But if you decide to proceed that way: to handle your concurrency issue could be through a retry-mechanism.
This is similar to how deadlocks are typically handled.
If the insertion fails because of violation of the unique primary key, just try again in PHP with a new key.
Your framework might have retry functions already. Otherwise it's easy to implement yourself.
We've got a web application that takes a uniquely generated workshop ID and calls a procedure (using php) and this procedure inserts it into a sql server table. This table has a clustered index on column workshopID and is set to unique.
This morning we had a user report that he got the following error code on his page:
Cannot insert duplicate key row in object 'dbo.table with unique
index 'ClusteredIndex-Wrkshp'. The duplicate key value is (Z9C1Am)
Obviously this suggests that I'm trying to insert Z9C1Am into the table and that value already exists---HOWEVER, when we do a simple lookup on that value, this value did not exist so I took the same stored procedure that the website code calls and used it to inserted Z9C1Am (using SSMS) into the table without any problem.
I can also get onto this application without any problems with this error; however, this same user called again and said he had the same issue (on the same computer) in another session. This time it had a different wordshopID in the error, but once again, this did not exist in the database.
I don't believe this has anything to do with inserting a duplicate key, rather, it is a phantom error.
Any suggestions on how to confirm this and how to track the actual error?
My gut reaction is this must be a browser related problem; however, all the code that interacts with the sql database is server side so this theory doesn't make much sense.
Thanks for the responses guys!
Greg's comment made me start looking harder at my dependencies and I have a join statement inside of my insert statement. The table I was joining to was supposed to contain unique values and approx. 25,000 of them are; however I had one set of duplicates causing the error.
This may be more of a design/logic type question, but what I'm attempting to do is poll a web service which returns about 15 "records" of foo. I then take those 15 records and attempt a SQL INSERT with them. If I poll again and get back 20 records, I attempt an INSERT only on the new 5 records.
My problem is on the first page load, when it performs the first poll, it will always return all the records. When I do the initial INSERT, of course I will typically get a bunch of "Violation of PRIMARY KEY constraint" errors. This is currently "by design", but I'm wondering if there's a better approach, or if this is an ok approach. Something about filling up the php error log repeatedly tells me it's not. :)
Is this resolved simply with a SQL-side Try/Catch?
Schema below:
If you want to avoid causing errors, you can check which of the posts already exists with an SQL statment like this
SELECT id FROM table WHERE id IN ([id list])
where id is your primary key column, table the table (surprise) and [id list] a comma separated list of the id´s you want to insert. Use the PHP function implode (documentation) to create one if you have the id´s in a list.
Then just exclude the rows with the returned id´s.
I'm developing an application where users will be importing a few thousand records from a text file. I have a unique constraint on 3 of the columns in my table, but when I attempt to import duplicate records I receive this error.
Error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '2013-06-01 15:25:41-2013-06-01 15:25:42-null' for key 'start_time'
It looks like CakePHP will stop attempting to insert data once an insert fails due to a constraint violation. Is there any way to simply have CakePHP ignore the constraint violation?
Thank you for your time.
It really depends on how your are importing your data and what RDBMS you are using.
If you are looping line-by-line over the text file and inserting data after each line, you could catch the exception and move on to the next line of your text file. Just remember to push the failed row into some kind of error log, so you'll be able to find which inserts failed. The bigger issue I see is that a thrown Exception might ruin your current transaction, and for mass data insertion you'll definitely want to wrap everything up in a singular transaction for best performance.
If you are using MySQL and CSV files, there's a LOAD DATA INFILE command you could explore using.
The problem is we are importing thousands of records at a time. Checking if a row exists before attempting to insert(I believe this is how CakePHP's unique validation works) will double the amount of queries if I were to try to save it row by row. I'm going to remove the unique constraint on the column and just insert all of the rows. After the new rows are inserted, I'm going to add a unique constraint to those columns and then remove the constraint. I think this will work well in my case because we plan on importing new records only once a month.
I am currently using RedBean for ORM in my models.
I need to insert data into a MySQL table where one of the columns is set to unique.
Currently, data is inserted via a form like so:
//Create
$object = R::dispense('object');
//Need to check if supplied name is a duplicate
$object->name = $name
$object->description = $description
//Save
R::store($object)
Now the problem is that if $name was not duplicated in the database, everything goes well. If it is a duplicate, I can catch the exception and get the SQL error code: 23000
If I echo the exception, I get:
[23000] - SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'abc' for key 'name_UNIQUE'
Another problem is that if I have multiple fields which are set to UNIQUE, the exception will only tell me the first field that's duplicated. Therefore, in order to get the next field, I need to make sure the user corrects the first duplicate and run the query again.
This seems to be quite inefficient. I need to be able to check if the user has inserted duplicate data in multiple fields while not executing too many SQL statements.
What are some best practices for doing this? Also, what's the best practice when it comes to returning whether the action was a success, or if on failure, why and which fields back to the controller?
Thank you :)
You must first perform a "select" to check that the unique entries do not already exist, there is no other way... If you do not want to perform multiple requests from you code, you can implement a stored procedure.
It allows you to perform multiple requests within the same transaction, and that limits the network overhead.
It will also allow you to manage multiple kinds of error (through error codes AFAIR), which will give you a way to identify clearly the problematic field.
Check out this link
Hope that helps !