Fastest way to combine thousands of arrays uniquely in PHP - php

Question

I am getting data from a database. Each result looks something like this
ASDF-1234-JKL-F1-STUFF
There are 50,000 results. Each one is being exploded
$exploded = explode('-',$dash_delimited_datum);
// $exploded = array('ASDF','1234','JKL','F1','STUFF');
I tried this:
$data = array();
while($row = mysql_fetch_array($result) ){
$i++;
if($i > 99999) {
break;
}
$data = array_merge($data,explode('-',$row[0]));
}
But I hit the server timeout of 5 minutes with it.
And this didn't work at all:
while($row = mysql_fetch_array($result) ){
$i++;
if($i > 99999) {
break;
}
$data_parts = explode('-',$row[0]);
foreach($data_parts as $value) {
$data = array_push(($data,$value);
}
}
Unexpectedly, this worked, taking "only" 9 seconds, but I wonder if I can make it even faster:
while($row = mysql_fetch_array($result) ){
$i++;
if($i > 99999) {
break;
}
$data = array_unique(array_merge($data,explode('-',$row[0])));
}
EDIT: I came up with an solution that I thought would be the best one, at 800ms
Note that I used a "closure" (the anonymous function) to remove numeric keys and I assumed it was a drag on speed. But actually, removing it caused the script to timeout at 30s.
$data=array();
while($row = mysql_fetch_array($result) ){
$i++;
if($i > 99999) {
break;
}
$data_parts = array_flip(array_filter(explode('-',$row[0]),
function($value) {
if(is_numeric($value)) {
return false;
} else return true;
}));
$data = array_merge($data,$data_parts);
}
$data = array_keys($data);
sort($data);
Conclusions:
Every fast answer used tricks involving the array keys, rather than values. And the difference between my best answer and the two very fast answers below seems to be their use of foreach inside the while loop to assign values directly to the main $data array. PHP function calls are supposedly expensive, and this example seems to prove that they really are. Both of the best answers gave me results in under 300 milliseconds. My best answer only worked fast when I filtered out numeric values, otherwise it ran into 30 second server timeout.
So, I guess if you are processing massive amounts of data, use constructs and not functions whenever you can.
Note about the (yes I know they're deprecated) mysql functions
One answer suggested that I use mysql_fetch_assoc instead of mysql_fetch_array. Actually, mysql_fetch_row is supposed to be "fastest", but the change made absolutely no difference in the speed of page loading with this data set (about 48,000 results). I also tried using mysql_result. The PHP docs say that it's slower when retrieving multiple rows, and it is a lot slower.
This took 6.27 seconds to load, versus about 0.27 seconds (270 milliseconds) for the similarly structured best answer.
$i=0;
while($data_parts = explode('-',mysql_result($result,$i,0)) ){
$i++;
if($i > 99999) {
break;
}
foreach($data_parts as $value) {
$data[$value] = 1;
}
}
$data = array_keys($data);

Answer 1

In order to speed the process, instead of using expensive functions to deal with arrays, using an associative array (hash) to ensure unique values should have the whole faster
$i = 0;
$hash = array();
while($row = mysql_fetch_array($result)) {
$i++;
if($i > 99999) {
break;
}
foreach (explode('-', $row[0]) as $s) {
$hash[ $s ] = 1;
}
}
This way, all strings are uniquely stored in an associative array (called hash).
The resulting array is $hash keys ( $data )
$data = array_keys( $hash );
print_r( $data );

Answer 2

How about this (I removed your counter, but you can add back in if its necessary):
$data = array();
$i = 0;
while($row = mysql_fetch_array($result) )
{
$data_parts = explode('-',$row[0]);
foreach($data_parts as $value)
{
if (!isset($data[$value]))
$data[$value] = $i++;
}
}
$data = array_flip($data);
I can't really benchmark on my computer, so if its slower than your implementations, let me know!

Answer 3

Try using mysql_fetch_assoc instead of mysql_fetch_array. mysql_fetch_array returns both numeric and associative indexes (effectively doubling the size of your array). In addition, try to use a little functions as possible within your while loop. For instance, if you iterate through 50,000 elements, and have 3 function calls in each iteration, thats 150,000 times the functions are called.
In addition, why not strip duplicates before even passing the result to the loop?
SELECT someField
FROM someTable
GROUP BY someField
HAVING COUNT(someField)>0
Once running that, run your loop
$data = array();
while($row = mysql_fetch_assoc($result) ){
$i++;
if($i > 99999) {
break;
}
$data[] = explode('-',$row[0]);
}