Why does `array_diff_ukey` call the compare function so many times? - php

Question

I executed the following code and its result made me confused!
I pass two arrays and a function named "myfunction" as arguments to the array_diff_ukey function. I see that myfunction is called 13 times (while it should be called at most 9 times). Even more amazing is that it compares the keys of the same array too! In both columns of the output, I see the key "e", while only the second array has it (the same is true for some other keys).
function myfunction($a,$b) {
echo $a . " ".$b."<br>";
if ($a===$b) {
return 0;
}
return ($a>$b)?1:-1;
}
$a1=array("a"=>"green","b"=>"blue","c"=>"red");
$a2=array("d"=>"blue","e"=>"black","f"=>"blue");
$result=array_diff_ukey($a1,$a2,"myfunction");
print_r($result);
Output:
a b
b c
d e
e f
a d
a e
a f
b d
b e
b f
c d
c e
c f
Array
(
[a] => green
[b] => blue
[c] => red
)
See it run on eval.in.
Why does the array_diff_ukey perform that many unnecessary calls to the compare function?

Answer 1

Nice question. Indeed the implemented algorithm is not the most efficient.
The C-source for PHP array functions can be found github. The implementation for array_diff_ukey uses a C-function php_array_diff which is also used by the implementations of array_udiff, array_diff_uassoc, and array_udiff_uassoc.
As you can see there, that function has this C-code:
for (i = 0; i < arr_argc; i++) {
//...
zend_sort((void *) lists[i], hash->nNumOfElements,
sizeof(Bucket), diff_key_compare_func, (swap_func_t)zend_hash_bucket_swap);
//...
}
...which means each input array is sorted using the compare function, explaining the first series of output you get, where keys of the same array are compared, and the first column can list other keys than the those of the first array.
Then it has a loop on the elements of the first array, a nested loop on the other arrays, and -- nested in that -- a loop on the elements of each of those:
while (Z_TYPE(ptrs[0]->val) != IS_UNDEF) {
//...
for (i = 1; i < arr_argc; i++) {
//...
while (Z_TYPE(ptr->val) != IS_UNDEF &&
(0 != (c = diff_key_compare_func(ptrs[0], ptr)))) {
ptr++;
}
//...
}
//...
}
Evidently, the sorting that is done on each of the arrays does not really contribute to anything in this algorithm, since still all keys of the first array are compared to potentially all the keys of the other array(s) with a plain 0 != comparison. The algorithm is thus O(klogk + nm), where n is the size of the first array, and m is the sum of the sizes of the other arrays, and k is the size of the largest array. Often the nm term will be the most significant.
One can only guess why this inefficient algorithm was chosen, but it looks like the main reason is code reusability: as stated above, this C code is used by other PHP functions as well, where it may make more sense. Still, it does not really sound like a good excuse.
A simple implementation of this (inefficient) array_diff_ukey algorithm in PHP (excluding all type checking, border conditions, etc) could look like this mimic_array_diff_ukey function :
function mimic_array_diff_ukey(...$args) {
$key_compare_func = array_pop($args);
foreach ($args as $arr) uksort($arr, $key_compare_func);
$first = array_shift($args);
return array_filter($first, function ($key) use($key_compare_func, $args) {
foreach ($args as $arr) {
foreach ($arr as $otherkey => $othervalue) {
if ($key_compare_func($key, $otherkey) == 0) return false;
}
}
return true;
}, ARRAY_FILTER_USE_KEY);
}
A more efficient algorithm would use sorting, but then would also take benefit from that and step through the keys of the first arrays while at the same time stepping through the keys of the other arrays in ascending order, in tandem -- never having to step back. This would make the algorithm O(nlogn + mlogm + n+m) = O(nlogn + mlogm).
Here is a possible implementation of that improved algorithm in PHP:
function better_array_diff_ukey(...$args) {
$key_compare_func = array_pop($args);
$first = array_shift($args);
$rest = [];
foreach ($args as $arr) $rest = $rest + $arr;
$rest = array_keys($rest);
uksort($first, $key_compare_func);
usort($rest, $key_compare_func);
$i = 0;
return array_filter($first, function ($key) use($key_compare_func, $rest, &$i) {
while ($i < count($rest) && ($cmp = $key_compare_func($rest[$i], $key)) < 0) $i++;
return $i >= count($rest) || $cmp > 0;
}, ARRAY_FILTER_USE_KEY);
}
Of course, this algorithm would need to be implemented in C if taken on board for improving array_diff_ukey, and to get a fair runtime comparison.
See the comparisons that are made -- on a slightly different input than in your question -- by the three functions (array_diff_ukey, mimic_array_diff_ukey and better_array_diff_ukey) on eval.in.

Answer 2

array_diff_ukey runs in two stages:
Sort the array keys
Compare key by key
This would probably explain why the callback is expected to return a sort value rather than a boolean "is equal".
I expect this is probably done for performance reasons, but if that's the case I would have thought that it can use this to say "well this key is bigger than all keys in the other array, so I shouldn't bother testing if these other, bigger keys are also bigger because they must be", but this doesn't seem to be the case: it compares them dutifully anyway.
I can only assume it's because the function cannot prove itself to be deterministic (and indeed in this case produces side-effects) so it can't be optimised like that. Perhaps array_diff_key (without user-defined function) does this optimisation just fine.
But anyway, that's what happens under the hood, and why you see more than just 9 comparisons. It could probably be made better in the core...