I'm developing an application where users will be importing a few thousand records from a text file. I have a unique constraint on 3 of the columns in my table, but when I attempt to import duplicate records I receive this error.
Error: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '2013-06-01 15:25:41-2013-06-01 15:25:42-null' for key 'start_time'
It looks like CakePHP will stop attempting to insert data once an insert fails due to a constraint violation. Is there any way to simply have CakePHP ignore the constraint violation?
Thank you for your time.
It really depends on how your are importing your data and what RDBMS you are using.
If you are looping line-by-line over the text file and inserting data after each line, you could catch the exception and move on to the next line of your text file. Just remember to push the failed row into some kind of error log, so you'll be able to find which inserts failed. The bigger issue I see is that a thrown Exception might ruin your current transaction, and for mass data insertion you'll definitely want to wrap everything up in a singular transaction for best performance.
If you are using MySQL and CSV files, there's a LOAD DATA INFILE command you could explore using.
The problem is we are importing thousands of records at a time. Checking if a row exists before attempting to insert(I believe this is how CakePHP's unique validation works) will double the amount of queries if I were to try to save it row by row. I'm going to remove the unique constraint on the column and just insert all of the rows. After the new rows are inserted, I'm going to add a unique constraint to those columns and then remove the constraint. I think this will work well in my case because we plan on importing new records only once a month.
Related
I'm fairly new to mysql and html/php.
This is the long, more in detail version, of my problem, I'll put a summary at the end of the post:
I've been developing a site in local using xampp, so I used phpmyadmin for the DDL part of development.
Problems started when in my php code I passed a variable from one page to another in an incorrect way it seems (this value should have been something like 0000000001 and instead it was 0). In this script, this value was used in an INSERT INTO query. Problem is that the value we're talking about was inserted in a column with a foreign key constraint, but this value didn't point to any primary key in the father table (I'm guessing this was the problem since this is where it all started going downhill). Suddenly the table in phpmyadmin was marked as "in use" and I started getting the error #1932 "table doesn't exist in engine". So I tried to drop it and discard tablespace: the DROP TABLE worked fine (the table disappear from the db view, even though it kept showing errno 1932), but I could not discard tablespace since it kept telling me that the table didn't exist. In mysql files I could see the .frm file gone, but not the .ibd. So I tried to delete it manually in order to be able to create this table once again (because if I tried to recreate it I was told to discard the tablespace first, but it's insane since I cannot discard it because I keep being told that the table does not exist). If I try to create again the table using the phpmyadmin UI I get the error #1005 that reference the error #150. So checking online I get the idea that I have to recreate the table exactly as it was, so from console I write down the CREATE TABLE query, including the foreign key constraint as shown in the "LATEST FOREIGN KEY ERROR" section of the error log, but instead I get the error #121.
SO, SUMMARIZING:
I want to make this table accessible to my code and myself again or create it again.
The table is marked in use and I can't access it.
If I try to drop it and discard tablespace I get error #1932.
If I try to recreate it I get errors because it already exixts, because I have to discard tablespace first or because I have a foreign key constraint incorrectly formed.
If I try to recreate it specifing the constraint I get an error because the constraint already exists.
This is the query I used to create the table specifing the constraint
CREATE TABLE `dbsmartcity`.`copertura` (
`Luogo` INT(10) UNSIGNED ZEROFILL NOT NULL ,
`Rete` INT(10) UNSIGNED ZEROFILL NOT NULL,
CONSTRAINT `copertura_ibfk_1` FOREIGN KEY (`Luogo`) REFERENCES `luoghi` (`IDLuogo`)
ON DELETE CASCADE ON UPDATE CASCADE
)
ENGINE = InnoDB DEFAULT CHARSET=utf8;
Yes, I have restarted MySQL.
Yup, I explicitly dropped the tablespace.
I would like to keep the remove/reinstall option as a last resort.
Sorry if I was not clear somewhere in this post. Please help me, my exam is near. Thanks a lot for your help.
I have application that receives multiple requests from external sources (invoices from point-of-sale units). It gets tens of requests per second, and some of those requests are the same (have same request body).
request data is transformed and saved to two associated tables (foreign key). if record already exists (queried by unique composite key), record is updated, otherwise record is added.
the problem is that sometimes if two requests with same body are received at the same time app throws exception that unique key already exists and it can't insert it:
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry
It's probably some kind of race condition in MySQL but can't figure it out
Whole process is wrapped in Laravel transaction. I tried setting different isolation levels. SERIALIZABLE resolves that issue but then I get lot of deadlock exceptions, and as I can see sometimes record is not saved at all.
This is simple, what happen here, you have some value declared as "UNIQUE" or "Primary Key" or something like that, and you are trying to insert again, some key restriction it's blocking the insert, this should be good, you avoid duplicate entries in your database, but what you need to do its check if the data what you are trying to insert exist before in the database, not all the columns, you should ask for your keys or key combination, I can not help you more if I don't know the data or the table...
I am trying to import well worked database to my phpmyadmin, though there is no duplicate entry for primary key. Since I have made auto increment for primary key, there is no chance for duplicate entry for it.I am using mysql 5.6.11 version.
there may be to reasons for that
1)let there are two tables in other table the first one table primary key is using as foreign key when you are importing the data the insert statements must be in order so that the table which primary key is using in another table must be insert data first
2)truncate the table and try import again if the same error occur then the first step follow that
It may not be the best solution but I think it may give you the solution if your database and the data is not too large. Separate your sql file into two pieces one for creating database ,table and relation and another for inserting data managing according to the foreign key. Once I have solved by this way this may help you.
You should not import .sql sql script simply, import create table script first then insert script later, as mentioned above truncanting table first then import script with the correct order if you occured at last 1 conflict duplicate entry problem, try truncanting your table again to fix the problem
Can there be any proper error handling be implemented into a SQL UNIQUE Constraint? So far i'm obviously getting: Error: Duplicate entry 'test1' for key 'username'
But i'd like to have it display a custom comment if possible
You should check in your app code before inserting this
row that no other row exists with the same username.
Then if it exists, show the error you want to the user.
Parsing this unique constraint error
(after you get it) is not a good idea.
So do it before, not after.
I am currently using RedBean for ORM in my models.
I need to insert data into a MySQL table where one of the columns is set to unique.
Currently, data is inserted via a form like so:
//Create
$object = R::dispense('object');
//Need to check if supplied name is a duplicate
$object->name = $name
$object->description = $description
//Save
R::store($object)
Now the problem is that if $name was not duplicated in the database, everything goes well. If it is a duplicate, I can catch the exception and get the SQL error code: 23000
If I echo the exception, I get:
[23000] - SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'abc' for key 'name_UNIQUE'
Another problem is that if I have multiple fields which are set to UNIQUE, the exception will only tell me the first field that's duplicated. Therefore, in order to get the next field, I need to make sure the user corrects the first duplicate and run the query again.
This seems to be quite inefficient. I need to be able to check if the user has inserted duplicate data in multiple fields while not executing too many SQL statements.
What are some best practices for doing this? Also, what's the best practice when it comes to returning whether the action was a success, or if on failure, why and which fields back to the controller?
Thank you :)
You must first perform a "select" to check that the unique entries do not already exist, there is no other way... If you do not want to perform multiple requests from you code, you can implement a stored procedure.
It allows you to perform multiple requests within the same transaction, and that limits the network overhead.
It will also allow you to manage multiple kinds of error (through error codes AFAIR), which will give you a way to identify clearly the problematic field.
Check out this link
Hope that helps !