I have three fields in my database: first name, lastname and email. Email is my primary key. I don't have any other fields in my database.
I need to find a mysql query which can delete duplicated primary keys and their values from the database leaving only one unique email in the database.
I use the following command to display all duplicated primary keys. It worked, but I need to delete all other duplicate entries and keep only one. I am using php programming.
SELECT *
FROM table_name
WHERE primarykey IN (
SELECT primarykey
FROM table_name
GROUP BY primarykey
HAVING count(primarykey) > 1
)
ORDER BY primarykey
Populate a temp table with the ones you want to keep, using GROUP BY, HAVING and MAX on the other columns. Then run your query which deletes too much, then put your copied ones back in. And then make it the actual PK so it doesn't happen again.
You cannot do this with just 1 query, as you will need to use limit and that one needs to be set hard. (You cannot say limit someColumn for example.)
$query = "select primarykey, count(primarykey) as count from table_name group by primarykey having count(primarykey) > 1"
$result = $mysqli->query($query);
while ($row = $result->fetch_assoc()) {
$query = "delete from table_name where primarykey = ? limit " . $row['count'];
$stmt = $mysqli->prepare($query);
$stmt->bind_param('s', $row['primarykey']);
$stmt->execute();
}
Email is not your primary key. Primary keys have a constraint on them where duplicates are not allowed. Your problem, by the way, is just an example of why you want to have an auto-incrementing numeric primary key on all tables. It seems too late for that.
One way to solve your problem is using temporary tables. The idea is to copy the data over, and tehn re-insert it:
create temporary table tmp_emails as
select email, firstname, lastname
from emails
group by email;
truncate table emails;
insert into emails(email, firstname, lastname)
select email, firstname, lastname
from tmp_emails;
Could you try this?
DELETE t1 FROM test t1, test t2
WHERE t1.email = t2.email
AND t1.fn > t2.fn
AND t1.ln > t2.ln;
Here are tests:
CREATE TABLE test
(
email varchar(100),
fn varchar(100),
ln varchar(100)
);
INSERT INTO test VALUES('a#b', 'f1', 'l1');
INSERT INTO test VALUES('a#b', 'f2', 'l2');
INSERT INTO test VALUES('c#d', 'f2', 'l2');
mysql> SELECT * FROM test;
+-------+------+------+
| email | fn | ln |
+-------+------+------+
| a#b | f1 | l1 |
| a#b | f2 | l2 |
| c#d | f2 | l2 |
+-------+------+------+
3 rows in set (0.00 sec)
mysql> DELETE t1 FROM test t1, test t2
-> WHERE t1.email = t2.email
-> AND t1.fn > t2.fn
-> AND t1.ln > t2.ln;
Query OK, 1 row affected (0.00 sec)
mysql> SELECT * FROM test;
+-------+------+------+
| email | fn | ln |
+-------+------+------+
| a#b | f1 | l1 |
| c#d | f2 | l2 |
+-------+------+------+
2 rows in set (0.00 sec)
ALTER TABLE table_name RENAME TO table_name_bak;
CREATE TABLE table_name AS SELECT * FROM table_name_bak LIMIT 0;
ALTER TABLE table_name ADD PRIMARY KEY (email);
INSERT IGNORE INTO table_name SELECT * FROM table_name_bak;
Related
I use mysql and php with phpmyadmin. I have major problem with a partition based counter that I wan't to improve but my knowledge on sql prevents me from doing that. Im struggling very much with this.
I want the duplicated data in my table to have a counter that adds a number after a value if this value gets a duplicated value and then restarts from 1 until a new value is met and so on. Here is what the final result should look like
---------------------------
1 | Josh-1
---------------------------
2 | Josh-2
--------------------------
3 | Josh-3
--------------------------
4 | Josh-4
--------------------------
5 | Fred-1
--------------------------
6 | Fred-2
--------------------------
7 | Fred-3
-------------------------
I had gotten help with this counter here before but it's not working as I wan't it to. Also when I have pressed the insert button in my form the table looks like this in phpmyadmin after I reload it
---------------------------
1 | Josh-1-1-1
---------------------------
2 | Josh-2
--------------------------
3 | Josh-3
--------------------------
4 | Josh-4
--------------------------
5 | Fred-1
--------------------------
6 | Fred-2
--------------------------
7 | Fred
-------------------------
Whats going on here? The code that I seek help with rewriting is this
UPDATE usermeta u1,
(SELECT
u1.`id`, CONCAT(u1.`name`,'-',ROW_NUMBER() OVER(PARTITION BY u1.`name` ORDER BY u1.`id`)) newname
FROM
usermeta u1 JOIN (SELECT `name` , COUNT(*) FROM usermeta GROUP BY `name` HAVING COUNT(*) > 1) u2
ON u1.`name` = u2.`name` ) u3
SET u1.`name` = u3.`newname`
WHERE u1.`id` = u3.`id`
Could this code be rewritten so it creates a table of numbered names and duplicates that looks like the first table example and work like it should in phpmyadmin ? All help is very much appreciated. Keep in mind that I am a struggling moderate sql user.
Possible solution - BEFORE INSERT trigger and additional MyISAM table with secondary autoincrement:
Working table
CREATE TABLE user (id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(127));
Additional table
CREATE TABLE user_index (id INT AUTO_INCREMENT,
name VARCHAR(127),
PRIMARY KEY (name, id)) ENGINE=MyISAM;
Trigger
CREATE TRIGGER insert_user_index
BEFORE INSERT ON user
FOR EACH ROW
BEGIN
DECLARE new_index INT;
INSERT INTO user_index (name) VALUES (NEW.name);
SET new_index = LAST_INSERT_ID();
DELETE FROM user_index WHERE name = NEW.name AND id < new_index;
SET NEW.name = CONCAT_WS('-', NEW.name, new_index);
END
Insert rows - the AI index is added to the name. Check the result.
INSERT INTO user (name) VALUES
('Josh'),
('Josh'),
('Fred'),
('Josh'),
('Fred'),
('Fred'),
('Josh');
SELECT * FROM user;
id | name
-: | :-----
1 | Josh-1
2 | Josh-2
3 | Fred-1
4 | Josh-3
5 | Fred-2
6 | Fred-3
7 | Josh-4
Look what is stored in additional table now.
SELECT * FROM user_index;
id | name
-: | :---
3 | Fred
4 | Josh
db<>fiddle here
If your working table user exists already, and it contains some data, then you'd create additional table and fill it with data using, for example,
CREATE TABLE user_index (id INT AUTO_INCREMENT,
name VARCHAR(127),
PRIMARY KEY (name, id)) ENGINE=MyISAM
SELECT MAX(SUBSTRING_INDEX(name, '-', -1) + 0) id,
SUBSTRING_INDEX(name, '-', 1) name
FROM user
GROUP BY 2;
https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=38f028cfe1c9e85188ab0454463dcd78
I need to understand how to do the following :
If a user has the diffPoints field changed (was 10 and became 20), then increment all upoints fields of all other users in the same group(s) as this user with the new value of diffPoints. I think I will do it via a trigger.
So first, a trigger will listen to any change in the "diffPoints" field in the users table.
second, I will get all users that are in the same group(s) as the specified user_id. I will do this by doing a query to get the groups in which the user is a member, and then use this result (i.e. groups) to get the other users in those groups via another query. And at last, I will update the upoints field for all these users.
First Query to get the diffPoints of a user :
select diffPoints
from
users
where
id=6
Second query to get all groups where this user exist :
select gp1.group_id
from
grpusrs as gp1
where
gp1.user_id = 6
Third query to get the users that are in the same group
SELECT
gp2.user_id
FROM
grpusrs as gp2
WHERE
gp2.group_id = gp1.group_id
Fourth query to update the upoints for all user_ids in query 3 :
UPDATE users increment upoints = (diffPoints in query1) WHERE users.id = (user_id from query 3)
Please advise in the following :
I need to know if the trigger solution is possible for this scenario.
Can I pass values between queries ?
Is there a better way to combine query 1,2,3 ?
What is the correct syntax for incrementing in query4 ?
You can do this in one multi table update statement (see https://dev.mysql.com/doc/refman/8.0/en/update.html)
given
drop table if exists us,ug;
create table us
(id int ,diffpoints int);
create table ug
(id int, uid int);
insert into us values
(1,10),(2,20),(3,30),(4,40),(5,50),(6,50);
insert into ug values
(1,1), (2,2),(2,3),(5,5),
(6,2), (6,6);
update us join
(
select us.id,ug.id grp
from us
join ug on ug.uid = us.id
where ug.id in
(select ug.id
from us
join ug on ug.uid = us.id and us.id = 2
)
) j on j.id = us.id
set diffpoints = 100;
Where the join works out all the users to be updated
+------+------------+
| id | diffpoints |
+------+------------+
| 1 | 10 |
| 2 | 100 |
| 3 | 100 |
| 4 | 40 |
| 5 | 50 |
| 6 | 100 |
+------+------------+
6 rows in set (0.00 sec)
I have a target table which crossed over 1M rows. Each time I will be getting 50K rows which may contain multiple duplicated entries. Hence I have decided to store CSV data into a temp table, then from temp_table to target_table by comparing rows between two tables...
If duplicated entries found append data from temp_table to target_table else Insert into the table... I am using partition here, so ON DUPLICATE key update is not working here.. in temp_table I am not using any KEYS
I have two tables which look like below
temp_table
Name | Type
John | Civil
John | Mech
target_table
Name | Type
John | Civil
When I run below query, I am getting an output of single row
UPDATE target_table JOIN temp_table
ON temp_table.Name = target_table.Name
SET target_table.Type = IF((LOCATE(temp_table.Type, target_table.Type) > 0)
target_table.Type,CONCAT(target_table.Type,',',temp_table.Type))
target_table
Name | Type
John | Civil
I am expecting output to be like below
target_table
Name | Type
John | Civil, Mech
May I know where it went wrong?
you should use a group_concat and use a subquery in join
UPDATE target_table
JOIN (
select name, group_concat(Type) grouped
from temp_table
group by name
) t ON t.Name = target_table.Name
SET target_table.Type = t.grouped
I suspect (but don't know for sure) and hopefully someone who does know will jump in and correct me, that an update join does not create a cartesian product in the way that a select would. As an attempted proof
truncate table temp_table;
insert into temp_table values
( 'John' , 'mech' ),
( 'John' , 'abc' );
truncate table target_table;
insert into target_table values
('john', 'civil', 9 );
UPDATE target_table JOIN temp_table
ON temp_table.Name = target_table.Name
set target_table.type = (concat(target_table.Type,',',temp_table.Type));
select * from target_table;
+------+------------+------+
| Name | Type | LOC |
+------+------------+------+
| john | civil,mech | 9 |
+------+------------+------+
1 row in set (0.00 sec)
note that abc from temp_table is ignored and mech is selected purely by chance.
if we change the order in temp_table
truncate table temp_table;
insert into temp_table values
( 'John' , 'abc' ),
( 'John' , 'mech' );
truncate table target_table;
insert into target_table values
('john', 'civil', 9 );
UPDATE target_table JOIN temp_table
ON temp_table.Name = target_table.Name
set target_table.type = (concat(target_table.Type,',',temp_table.Type));
select * from target_table;
we get
+------+-----------+------+
| Name | Type | LOC |
+------+-----------+------+
| john | civil,abc | 9 |
+------+-----------+------+
1 row in set (0.02 sec)
where abc is picked purely by chance.
In my view the safest way to do this is on a row by row basis ie a cursor.
so, i have done a big mistake when i designing my table for salesreport that looks like this
+----+------------+--------------+-------+
| id | company_id | company_code | value |
+----+------------+--------------+-------+
| 1 | 0 | 67 | 100 |
| 2 | 0 | 55 | 200 |
+----+------------+--------------+-------+
i just recently notice it and add new column called company_id which is in the company table that looks like this
+----+--------------+------+
| id | company_code | name |
+----+--------------+------+
| 1 | 55 | XX |
| 2 | 67 | XA |
+----+--------------+------+
in the past i create relationship with company_code since i thought it will always unique but not auto increment, the code is created manually from company list in existing record.
i then realise that it will be better to create a relationship between table by using id so instead of using company_code it should be company_id on my salesreport table that pointing to id column in company table
and now there is more than a thousand record that already in mysql database that referencing relationship using company_code and i want to know is there one times mysql query that i can run to fix it?
and i come up with this kind of query
UPDATE salesreport SET company_id = '1' WHERE company_code = '67';
but i think since both tables company and salesreport already had a relationship why can't it just like this
UPDATE salesreport SET company_id = company.id WHERE company_code = company.company_code;
but i don't think it will be work, it needs more query to know that i selecting company table and then match salesreport.company_code with company.company_code and if it is match then set salesreport.company_id with company.id
well i think that is how it goes... but i have no idea what is the query to do just that.. so maybe someone can help me and provide a lazy-elegant solution to this.
thank you by the way.
Here is a solution to your request:
UPDATE salesreport
LEFT JOIN company ON company.company_code = salesreport.company_code
SET salesreport.company_id = company.id;
I have created a SQLFiddle for you to test the results: http://rextester.com/HNJ85353
HerŅ is a full test case:
CREATE TABLE IF NOT EXISTS p1929_salesreport (id INTEGER PRIMARY KEY AUTO_INCREMENT,
company_id INTEGER,
company_code INTEGER,
value INTEGER);
INSERT INTO p1929_salesreport (company_id, company_code, value) VALUES (0, 67, 100);
INSERT INTO p1929_salesreport (company_id, company_code, value) VALUES (0, 55, 200);
CREATE TABLE IF NOT EXISTS p1929_company (id INTEGER PRIMARY KEY AUTO_INCREMENT,
company_code INTEGER,
name TEXT);
INSERT INTO p1929_company (company_code, name) VALUES (55, "XX");
INSERT INTO p1929_company (company_code, name) VALUES (67, "XA");
/* before changes */
SELECT * FROM p1929_company;
SELECT * FROM p1929_salesreport;
/* actual query */
UPDATE p1929_salesreport
LEFT JOIN p1929_company ON p1929_company.company_code = p1929_salesreport.company_code
SET p1929_salesreport.company_id = p1929_company.id;
/* after changes */
SELECT * FROM p1929_salesreport;
DROP TABLE p1929_salesreport;
DROP TABLE p1929_company;
Update with a join to your other table
UPDATE salesreport s
LEFT JOIN company c on s.company_code = c.company_code
SET s.company_id = c.id;
You could update using an inner join eg:
UPDATE salesreport s
INNER JOIN company c ON s.company_code = c.company_code
set s.company_id = c.id
MERGE into salesreport S
USING(select id, company_code, name from Company) C
ON(S.company_code=C.company_code)
WHEN MATCHED THEN
UPDATE
SET S.company_id = C.id
you can try this. this will update Salesreport table using company table when both company code matches.
Clarification of what I'm trying to accomplish:
Update "user name", if there is a record with the same e-mail but no name. if user already has a name, then don't make any changes
I have a website where all users can be "subscribers, entrepreneurs or investors". If a subscriber (of whom I have previously only asked for email) chooses to upload a business idea then that person will probably use the same e-mail address as before only this time adding a name. So I'd like to INSERT INTO, and if e-mail already exists - add the name, but only if there is not a name there already (so that a person cannot simply over write somebody else's details).
I've gotten this far:
mysql_query("
INSERT INTO users
(email, name)
VALUES
('" .$epost. "', '" .$namn. "')
ON DUPLICATE KEY UPDATE
name=VALUES(name) -->[if name == '', else do nothing...]
");
Now it replaces the current name with a new one if different.
I searched "on duplicate update if field empty"
and found:
http://forums.aspfree.com/sql-development-6/on-duplicate-key-update-but-only-if-value-isn-t-482012.html (merging?)
conditional on duplicate key update (closer, but I don't want to update it if it differs, only if the field is empty.)
http://bytes.com/topic/php/answers/914328-duplicate-key-update-only-null-values (if the input is blank.. well it's not)
http://boardreader.com/thread/Insert_on_duplicate_key_update_only_if_c_can8Xachr.html (the layout of this page got me lost)
http://www.digimantra.com/tutorials/insert-update-single-mysql-query-duplicate-key/ (it simply updates)
I believe this might be kind of what I'm after (?) http://thewebfellas.com/blog/2010/1/18/conditional-duplicate-key-updates-with-mysql but I haven't managed to make it work
This code:
mysql_query("UPDATE users SET name='".$namn."'
WHERE email='".$epost."' AND name =''");
updates the name, only if it's previously null, and that's what I'm after however it does not insert a new record if email doesn't already exist.
My table
So I tried this:
mysql_query("
INSERT INTO users
SELECT email, 'victoria' FROM users
WHERE email='victoria#hejsan.se' ON DUPLICATE KEY UPDATE name = 'victoria'
");
and I tried this:
mysql_query("
INSERT INTO users
SELECT email, 'yay' from users
WHERE email='victoria#hejsan.se'
ON DUPLICATE KEY
UPDATE name = values(name)
");
from #Fluffeh 's answer
but nothing happens. Did i misinterpret the answer?
It just hit me that if someone already has an account then they should also have a password already, and so I'll just ask them to verify their password, and make sure I do an AJAX call while they insert their e-mail to see if the e-mail is already registered :) And so then this is no longer an issue, and I think that is the solution I'll go with. !)
The only way that you could use the insert .... on duplicate key... syntax to solve your issue would be if name and email made a composite key - and I think you would be better off using an auto_increment as a primary key.
You might have to put a little logic into the PHP to do a check first, then insert or update - or write a function to do that same test for you - but neither will be a simply query that you can just fire off.
Edit: Not sure if this is what you want to do, but I think that the best solution for your requirements is to actually use two tables in a one to many relationship.
Table-Users
id | email | name
create table users(
id int(10) not null auto_increment,
email varchar(100),
name varchar(100),
primary key(email, name)
);
Table-Ideas
id | userID | idea
create table users(
id int(10) not null auto_increment primary key,
userID int(10) not null,
idea text
);
With the primary key on the table, you can safetly do an insert... duplicate... without worrying about over-writing folks. The second table will however allow you to have the ideas stored safetly locked to the user, and let you have a number of ideas per user. As the relationship is the users.id to ideas.userID you won't lose who owns it even if their details are updated.
Edit: (aka, ZOMG facepalm)
$query="
update users
set name='".$userName."'
where
email='".$userEmail."'
and name is null";
Edit 2: (aka, wipes brow)
insert into users
select email, '".$namn."' from users where email='".$epost."'
on duplicate key
update name = values (name);
and here it is working:
mysql> create table test1 (myName varchar(10) unique, myEmail varchar(10));
Query OK, 0 rows affected (0.02 sec)
mysql> insert into test1 values ('Tom','something');
Query OK, 1 row affected (0.01 sec)
mysql> insert into test1 values('Nick',null);
Query OK, 1 row affected (0.00 sec)
mysql> select * from test1;
+--------+-----------+
| myName | myEmail |
+--------+-----------+
| Tom | something |
| Nick | NULL |
+--------+-----------+
2 rows in set (0.00 sec)
mysql> insert into test1 select myName, myEmail from test1
where myName='Tom' on duplicate key update myEmail = values (myEmail);
Query OK, 0 rows affected (0.00 sec)
Records: 1 Duplicates: 0 Warnings: 0
mysql> select * from test1;
+--------+-----------+
| myName | myEmail |
+--------+-----------+
| Tom | something |
| Nick | NULL |
+--------+-----------+
2 rows in set (0.00 sec)
mysql> insert into test1 select 'Tom', myEmail from test1
where myName='Tom' on duplicate key update myEmail = values (myEmail);
Query OK, 0 rows affected (0.00 sec)
Records: 1 Duplicates: 0 Warnings: 0
mysql> select * from test1;
+--------+-----------+
| myName | myEmail |
+--------+-----------+
| Tom | something |
| Nick | NULL |
+--------+-----------+
2 rows in set (0.00 sec)
mysql> insert into test1 select myName, 'Something Else' from test1
where myName='Tom' on duplicate key update myEmail = values (myEmail);
Query OK, 2 rows affected, 1 warning (0.01 sec)
Records: 1 Duplicates: 1 Warnings: 1
mysql> select * from test1;
+--------+------------+
| myName | myEmail |
+--------+------------+
| Tom | Something |
| Nick | NULL |
+--------+------------+
2 rows in set (0.00 sec)
mysql> insert into test1 select myName, null from test1
where myName='Nick' on duplicate key update myEmail = values (myEmail);
Query OK, 0 rows affected (0.00 sec)
Records: 1 Duplicates: 0 Warnings: 0
mysql> select * from test1;
+--------+------------+
| myName | myEmail |
+--------+------------+
| Tom | Something |
| Nick | NULL |
+--------+------------+
2 rows in set (0.00 sec)
mysql> insert into test1 select myName, 'yay' from test1
where myName='Nick' on duplicate key update myEmail = values (myEmail);
Query OK, 2 rows affected (0.01 sec)
Records: 1 Duplicates: 1 Warnings: 0
mysql> select * from test1;
+--------+------------+
| myName | myEmail |
+--------+------------+
| Tom | Something |
| Nick | yay |
+--------+------------+
2 rows in set (0.00 sec)
Edit 3: Try this for your $query
insert into table1 select coalesce(email,'".$epost."') as email, coalesce(name,'".$namn."') as name from table1
where email='".$epost."' on duplicate key update name = values (name);
If you want to update a field that is already exist, you can do this as;
-- MySQL Reference
INSERT INTO table (a,b,c) VALUES (1,2,3),(4,5,6)
ON DUPLICATE KEY UPDATE c=VALUES(a)+VALUES(b);
-- so your code
INSERT INTO users
(email, name)
VALUES
('" .$epost. "', '" .$namn. "')
ON DUPLICATE KEY UPDATE email=VALUES(email), name=VALUES(name)
http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html
UPDATE
At this point, you need "MySQL conditional insert". Let's try it;
CREATE TABLE IF NOT EXISTS `users` (
`name` varchar(50) NOT NULL,
`email` varchar(255) NOT NULL,
UNIQUE KEY `email` (`email`)
)
INSERT INTO users (name, email) VALUES('kerem', 'qeremy[atta]gmail[dotta]com');
INSERT INTO users (name, email) VALUES('test', 'test#example.com');
>> 2 rows inserted. ( Query took 0.0005 sec )
-- this is your part (regarding email field is UNIQUE, so WHERE email='search email')
INSERT INTO users (name, email)
SELECT 'test', 'test#example.com' FROM DUAL WHERE NOT EXISTS (
SELECT * FROM users WHERE email = 'test#example.com' LIMIT 1
);
>> 0 rows inserted. ( Query took 0.0003 sec )
https://www.google.com/search?q=conditional+insert+mysql
http://allurcode.com/2011/04/15/mysql-conditional-insert/ (which is useful search result)
UPDATE 2
If this does not work for you, then you can do your work like (just an input);
mysql_query("INSERT IGNORE INTO users (name, email) VALUES('$name', '$email')");
if (!mysql_insert_id()) {
mysql_query("UPDATE users SET name='$name' WHERE name='' AND email='$email'");
}
Or;
mysql_query("UPDATE users SET name='$name' WHERE name='' AND email='$email'");
if (mysql_affected_rows() < 1) {
mysql_query("INSERT INTO users (name, email) VALUES('$name', '$email')");
}
I'd like to suggest this solution using the control function IFNULL(exp1,exp2), defined as "if exp1 is not null, returns exp1; if exp1 is null, returns exp2".
http://dev.mysql.com/doc/refman/5.7/en/control-flow-functions.html
INSERT INTO users (name, email) VALUES ('$name','$email')
ON DUPLICATE KEY UPDATE name = IFNULL(name, VALUES(name))
Of course you must be beyond this somehow, but it helped me.