Find duplicate rows in mysql simplified example - php

Question

Confession: a mysql newb requires simple example to locate duplicate rows in somewhat large table. I have searched for and read many other threads with similar titles, but the examples are so complex that I cannot apply them to my basic situation.
A MySQL table has only 5 fields, but there are hundreds of rows. I wish to locate duplicate rows -- I know there is one for sure and wonder if there are others.
Example Row: (rel_id is auto-incrementing, primary key field)
'rel_id' => 1
'host' => 17
'host_type' => 'client'
'rep' => 7
'rep_type => 'cli_mgr'
My approach was to:
1. Read entire table into mysql query
2. row-by-row compare the 4 data fields to those of previous ("done") rows
3. after comparing a "new" row, append it to array of "done" rows
Here is what I have tried. I am sure that there must be a much simpler solution. You will see that I am bogged down in trying to append the "new" row to the array of "done" rows:
$rRels = mysql_query("SELECT * FROM `rels`");
$a = array();
$e = array();
$c1 = 0;
$c2 = 0;
While ($r = mysql_fetch_assoc($rRels)) {
$i = $r['rel_id'];
$h = $r['host'];
$ht = $r['host_type'];
$r = $r['rep'];
$rt = $r['rep_type'];
foreach($a as $row) {
$xh = $row['host'];
$xht = $row['host_type'];
$xr = $row['rel'];
$xrt = $row['rel_type'];
if (($h==$xh) && ($ht==$xht) && ($r==$xr) && ($rt==$xrt)) {
echo 'Found one<br>';
$e[] = $r;
}
$c2++;
}
$a = array_merge(array('rel_id'=>$i, 'host'=>$h, 'host_type'=>$ht, 'rep'=>$r, 'rep_type'=>$rt), $a);
$c1++;
}
echo '<h3>Duplicate Rows:</h3>';
foreach ($e as $row) {
print_r($row);
echo '<br>';
}
echo '<br><br>';
echo 'Counter 1: ' . $c1 . '<br>';
echo 'Counter 2: ' . $c2 . '<br>';

Answer 1

This should do the trick:
SELECT COUNT(*) as cnt, GROUP_CONCAT(rel_id) AS ids
FROM rels
GROUP BY host, host_type, rep, rep_type
HAVING cnt > 1
any "duplicate" records will have a cnt > 1, and the group_concat will give you the ids of the duped records.

Answer 2

Pure no-php solution : to make the copy of the old table (named oldTable) , with no data
create table newTable like oldTable;
Modify the structure to prevent duplicates and add unique key over all 5 columns.
alter table newTable add unique index(rel_id,host,host_type,rep,rep_type );
Then whith sql query copy the rows from oldTable
insert IGNORE into newTable select * from oldTable
In newTable you have only the unique data.
Another option is group by, if you will get the number of duplicate rows use
select concat_ws('_',rel_id,host,host_type,rep,rep_type) as str, count(*)
from oldTable
group by str

Answer 3

You can this query to find all the duplicate rows. Hopefully, it should be easy integrating in your PHP code.
// This will give you all the duplicates
// Self join where all the columns have the same values but different primary keys
SELECT *
FROM rels t1, rels t2
WHERE t1.rel_id != t2.rel_id
AND t1.host = t2.host
AND t1.host_type = t2.host_type
AND t1.rep = t2.rep
AND t1.rep_type = t2.rep_type

Answer 4

Finding duplicates is more easily done in SQL than in PHP.
SELECT GROUP_CONCAT(rel_id)
FROM rels
GROUP BY host, host_type, rep, rep_type HAVING COUNT(rel_id)>1;
This will show the groups of rel_id that point to identical records. The HAVING COUNT(rel_id)>1 clause allows us to skip unduplicated records.