i am stuck in a simple update query.
i have a table say tabble1 containing 'name' and 'phone_no' column. Now when i upload csv file containing list of name and contact numbers, i want to update name of duplicate number with previous one. For ex. i have a row containing 'max' '8569589652'. now when i upload same number with another name say 'stela' '8569589652' then stela shuld get updated to max.
for this purpose i created another table say table2. then i collected all duplicate entries from table1 into table2. after that updated new entry with previous name.
following are my queries:
to collect all duplicate entries:
INSERT INTO table2 SELECT phone_no,name FROM table1
GROUP BY phone_no HAVING COUNT(*)>1;
to update duplicate entries in table1:
UPDATE table1.table2 SET table1.name=table2.name
WHERE table1.phone_no=table2.phone_no ;
My problem is when i run these two query it is taking tooo much of time. It is taking ore than half an hour to upload csv file of 1000 numbers.
Please suggest me optimize query to upload csv in less time.
does speed of uploading matters with size of database..
please help.
thanks in advance.
Here are the steps from the link I suggested.
1) Create a new temporary table.
CREATE TEMPORARY TABLE temporary_table LIKE target_table;
2) Optionally, drop all indices from the temporary table to speed things up.
SHOW INDEX FROM temporary_table;
DROP INDEX `PRIMARY` ON temporary_table;
DROP INDEX `some_other_index` ON temporary_table;
3) Load the CSV into the temporary table
LOAD DATA INFILE 'your_file.csv'
INTO TABLE temporary_table
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
(field1, field2);
4) Copy the data using ON DUPLICATE KEY UPDATE
SHOW COLUMNS FROM target_table;
INSERT INTO target_table
SELECT * FROM temporary_table
ON DUPLICATE KEY UPDATE field1 = VALUES(field1), field2 = VALUES(field2);
5) Remove the temporary table
DROP TEMPORARY TABLE temporary_table;
You can update duplicate entries of "phone no" like below.
INSERT INTO table2 (phone_no,name)
VALUES
('11111', aaa),
('22222', bbb),
('33333', cccc),
ON DUPLICATE KEY UPDATE
phone_no = VALUES(phone_no);
Dump your CSV file in a temp table.
Then aply merge statement simply
Merge
AS MAIN
USING
AS TEMP
On
MAIN.CONTACT_NO=TEMP.CONTACT_NO
WHEN MATCHED THEN UPDATE
MAIN.NAME=TEMP.NAME;
IF YOU WANT TO INSERT NON MATCHING RECORD USE THIS
WHEN NOT MATCHED
THEN INSERT
(NAME,
CONTACT_NO)
VALUES
(
TEMP.NAME,
TEMP.CONTACT_NO
);
Please not that merge command must end with ';'
I have used ';' after upadate remove that and add the below part and end the whole merge with ';'
Hope this helps
Please update if any more help needed.
Related
there is my code sample that deletes data from queue_db,I need code which inserts queue_db values to staff_db table.enter image description here
You can use SQL query
Copy all record one table to another
insert into staff_db select * from queue_db
This will transfer the data one table to another, later you can delete records conditionally from first table
If the columns names are different then
Copy all record one table to another with column name alias
insert into staff_tb(column1, column2, column3) select c as column1, b as column2, d as column3 from queue_db
where c,b,d are the columns in the second table
If you want to filter it with condition then add where at the end of the query with <tablename.columnname> = <value>
Copy all record one table to another without duplicates
To prevent duplicates, you can use this answer Prevent Duplicate I don't want repeat the same answer here
If you want copy those data into fresh new table like doing a backup then
Copy all record one table to a fresh new table
select * into newtable from queue_db
Then you run delete query for table which you copied from
Delete all records from a table
delete from queue_db
I need to add a CSV file in a database. But after I add one file, some weeks later I need to add a new update of the file. Problem is: Duplicate rows are added.
I don't have an ID for rows so I have to see if they are the same with 'City', 'Address' and 'Location Name'. Only if the 3 are matching then we dont put the new row in database.
I tried IGNORE but it seems to only work whit an ID as primary key (and I don't have primary key).
I also read a 'multiple primary key' thread but I did not succeed to create it.
My actual code is (Codeigniter):
$query = $this->db->query('
LOAD DATA INFILE "'.$path.'fichier/'.$fichier.'"
INTO TABLE location FIELDS TERMINATED BY ";"
LINES TERMINATED BY "'.$os2.'"
IGNORE 1 LINES ('.$name[1].','.$name[2].','.$name[3].','.$name[4].','.$name[5].','.$name[6].','.$name[7].','.$name[8].','.$name[9].','.$name[10].','.$name[11].','.$name[12].','.$name[13].','.$name[14].','.$name[15].','.$name[16].','.$name[17].','.$name[18].','.$name[19].')');
I would say, best would be to load your updated CSV in a staging table. Once all data loaded, do a LEFT JOIN with your actual table and find out all the new records and then insert only those new records to your main table (OR) Once all data loaded, flush all the data in main table with this new staging table.
Per your comment:
Yes, one you have loaded data to new table perform a LEFT JOIN with your main table (something like below)
select staging_table.id
from staging_table
left join main_table on staging_table.id = main_table.id
where main_table.id is null;
If someone want to know how I finally succeed:
I created a unique in phpmyadmin. And then I used a IGNORE on my request.
ALTER TABLE location
ADD CONSTRAINT iu_location UNIQUE( col1, col2, col3 );
I know this has been asked before, but I'm not a coder, and cannot figure it out from other similar posts. I've spent over 5 hours trying to figure this out with great unsuccess :( So I ask for your help.
1) Prevent Duplicates
I have a PHP script that writes to DB. Here is the code:
$sql = "INSERT INTO results (total, size, persq, strip, material, region)
VALUES ('$total', '$size', '$persq', '$strip', '$material', '$region')";
I want to prevent duplicate rows based on TOTAL and SIZE columns. So if a new entry matches value in TOTAL and SIZE, do not enter new row.
2) Delete Duplicates
I want to delete ALL existing douplicate rows from DB, also based on TOTAL and SIZE columns.
If row contains duplicates in both TOTAL and SIZE, delete entire row.
How do I do this?
PS - I've read that I can use SQL IGNORE command to prevent futue duplicates - example (i've tryed to structure it to work for my situation:
INSERT IGNORE INTO results ...;
would something like this work? If so please help me structure it (i'm new to PHP and MySQL).
Big thanks in advance.
I think the easiest way to remove the duplicates is to use a CTAS (Create Table As Select) statement to create a temporary table for your data. Using group by, you can remove the duplicates. MySQL is 'smart' enough to just pick any value for the other fields from one of the rows that match the group.
/* De-duplicate and copy all the data to a temporary table. */
CREATE TABLE Temp AS
SELECT * FROM results
GROUP BY total, size;
/* Delete all data from your current table. Truncate is faster but more dangerous. */
DELETE FROM results; /* TRUNCATE results; */
/* Insert the de-duplicated data back into your table. */
INSERT INTO results
SELECT * FROM Temp;
/* Drop the temporary table. */
DROP TABLE Temp;
After that you can add a unique constraint for total,size to prevent new duplicates.
ALTER TABLE results
ADD UNIQUE results_uni_total_size (total, size);
If you have duplicate rows where EVERY column has a duplicate value, the easiest thing to do is crate a new table and import all of the rows using a group by on every column. First create a new table with each column set as a unique key:
CREATE TABLE newresults total INT NOT NULL, size ...
UNIQUE KEY (total, size, presq, strip, material, region)
Then push clean values into the new table:
INSERT INTO newresults (total, size, persq, strip, material, region) SELECT total, size, persq, strip, material, region FROM RESULTS GROUP BY total, size, persq, strip, material, region
That will give you a clean data set. The final thing you'll have to do is drop the old table and rename newresults to results:
DROP TABLE results;
RENAME TABLE mydatabase.newresults TO mydatabase.results
Hope that helps...
I have a large database with many tables with several thousand records.
I have done some Excel work to identify some records / rows I want to delete from one of my large tables, as when I tried to do the query within phpmyadmin, the table kept locking as the query was too big.
Anyway.... Long story short.
I now have a list of 1500 records I need to delete from one of my tables.
Is there a way to "paste" these 1500 values into a query, so I can bring back the matching records, select them all at once and delete them?
Obviously, I dont want to do this manually one at a time.
So the query I have in mind is something like this:
Find any records which match these IDs (WHERE ID = )
Paste in list of IDs from Ms Excel
Results returned
Can select all rows and delete
Any tips?
Thanks
Just use the Keyword "IN" in your query with your list of value. Like :
Select Name
From Users
Where ID IN (1,2,3,4 .....) ;
There is more than one way to do this.
First:
You can directly paste the list of coma separated IDs in the where clause.
DELETE FROM tablename WHERE ID IN (1,2,3,4);
It you get error 'Packet Too Large' you can increase max_allowed_packet
The largest possible packet that can be transmitted to or from a MySQL
5.0 server or client is 1GB.
Second:
You can export you Excel file to csv file and load the data to a temp table then delete fro the table using the tmp table as reference
LOAD DATA INFILE 'X:\filename.csv'
INTO TABLE tmptable
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n';
DELETE t1
FROM tablename t1
JOIN tmptable t2 ON t1.ID = t2.ID;
Reference: MySQL LOAD DATA INFILE Syntax
Don't forget to remove your tmp table.
i have a contactnumber column in mysql database. In contactnumber column there are more than 20,000 entries. Now when i upload new numbers through .csv file, i dont want duplicate numbers in database.
How can i avoid duplicate numbers while inserting in database.
I initially implemented logic that checks each number in .csv file with each of the number in database.
this works but takes lot of time to upload .csv file containing 1000 numbers.
Pleae suggest how to minimize time required to upload .csv file while not uploading duplicate values.
Simply add a UNIQUE constraint to the contactnumber column:
ALTER TABLE `mytable` ADD UNIQUE (`contactnumber`);
From there you can use the IGNORE option to ignore the error you'd usually be shown when inserting a duplicate:
INSERT IGNORE INTO `mytable` VALUES ('0123456789');
Alternatively, you could use the ON DUPLICATE KEY UPDATE to do something with the dupe, as detailed in this question: MySQL - ignore insert error: duplicate entry
If your contactnumber should not be repeated then make it PRIMARY or at least a UNIQUE key. That way when a value is being inserted as a duplicate, insert will fail automatically and you won't have to check beforehand.
The way I would do it is to create a temporary table.
create table my_dateasyyyymmddhhiiss as select * from mytable where 1=0;
Do your inserts into that table.
and then query out the orphans on the between mytable and the temp table based on contactnumber
then run an inner join query between the two tables and fetch out the duplicate for your telecaller tracking.
finally drop the temporary table.
Thing that this does not address are duplicates within the supplied file (don't know if that would be an issue in this problem)
Hope this help
If you don't want to insert duplicate values in table and rather wants to keep that value in different table.
You can create trigger on table.
like this:
DELIMITER $$
CREATE TRIGGER unique_key BEFORE INSERT ON table1
FOR EACH ROW BEGIN
DECLARE c INT;
SELECT COUNT(*) INTO c FROM table1 WHERE itemid = NEW.itemid;
IF (c > 0) THEN
insert into table2 (column_name) values (NEW.itemid);
END IF;
END$$
DELIMITER ;
I would recommend this way
Alter the contactnumber column as UNIQUE KEY
Using phpmyadmin import the .csv file and check the option 'Do not abort on INSERT error' under Format-Specific Options before submitting