How do I find fuzzy duplicates from this php array? - php

Question

Before I add clarification, here is some pseudo data. The array I need to iterate is like this:
$ipBodies = array(
'1.2.3.4' => array(
array('id' => 1, 'body' => 'asdfasdfasdf_X'),
array('id' => 2, 'body' => 'asdfasdfasdf_Y'),
array('id' => 3, 'body' => '123456789_X'),
array('id' => 4, 'body' => '123456789_Y'),
),
'5.6.7.8' => array(
array('id' => 13, 'body' => 'foobarbaz_X'),
array('id' => 14, 'body' => 'foobarbaz_Y'),
array('id' => 15, 'body' => 'adsflkjlsdfjlkjlkasdfj'),
array('id' => 16, 'body' => 'foobarbaz_Z'),
),
);
So from this sample data, you can see there are two sets of unique 'fuzzy duplicates' in the 1.2.3.4 array, and only 1 set of 'fuzzy duplicates' in the 5.6.7.8 array.
In the real data, everything is scaled up. The main array will have hundreds of ip addresses, and those arrays could have hundreds of members. Also the body section is larger in the real data.
I've considered that I need to run through each ip address array and create a new array of every combination to a new array, say $pairs, then run similar_text (seems to work well for this) on those to find duplicates, but creating these sets of pairs will be expensive I believe. I think the $pairs array count would end up being the factorial of the count of the array, which could become enormous as the array size increases.
I'm thinking I'd like to end up with an array $dupes that (based on the sample data above) should look like this:
$dupes = array(
'1.2.3.4' => array(
array('1', '2'),
array('3', '4'),
),
'5.6.7.8' => array(
array('13', '14', '16'),
),
);
I really just need some help and advice here so I can start solving the problem. God I hope my explanation made sense. If it didn't, let me know and I'll clarify.

Answer 1

If possible, I recommend using levenshtein instead of similar_text because it's a faster algorithm.
The complexity of the algorithm is O(m*n), where n and m are the
length of str1 and str2 (rather good when compared to similar_text(),
which is O(max(n,m)**3), but still expensive).
The code below uses an associative array to put each element into buckets where the ip['body'] has levensthein distance of < 2 (which means matches within the same bucket will have at most 1 different character, change as needed). Once all elements have been placed into their respective buckets every bucket with only 1 element is discarded.
$ipBodies = array(
'1.2.3.4' => array(
array('id' => 1, 'body' => 'asdfasdfasdf_X'),
array('id' => 2, 'body' => 'asdfasdfasdf_Y'),
array('id' => 3, 'body' => '123456789_X'),
array('id' => 4, 'body' => '123456789_Y'),
),
'5.6.7.8' => array(
array('id' => 13, 'body' => 'foobarbaz_X'),
array('id' => 14, 'body' => 'foobarbaz_Y'),
array('id' => 15, 'body' => 'adsflkjlsdfjlkjlkasdfj'),
array('id' => 16, 'body' => 'foobarbaz_Z'),
),
);
$counts = [];
foreach($ipBodies as $groupName => $group) {
$counts[$groupName] = [];
foreach($group as $key => $ip) {
foreach($counts[$groupName] as $countGroup => $groupCount) {
if(levenshtein($ip['body'],$countGroup) < 2) {
$counts[$groupName][$countGroup][] = $ip['id'];
continue 2;
}
}
$counts[$groupName][$ip['body']] = [$ip['id']];
}
}
//remove elements that appear just once
foreach($counts as $groupName => &$groupCounts) {
foreach($groupCounts as $k => &$v) {
if(count($v) < 2) {
unset($counts[$groupName][$k]);
}
}
$counts[$groupName] = array_values($groupCounts);
}
print_r($counts);
Output
Array
(
[1.2.3.4] => Array
(
[0] => Array
(
[0] => 1
[1] => 2
)
[1] => Array
(
[0] => 3
[1] => 4
)
)
[5.6.7.8] => Array
(
[0] => Array
(
[0] => 13
[1] => 14
[2] => 16
)
)
)