Finding and removing outliers in PHP - php

Question

Suppose I sample a selection of database records that return the following numbers:
20.50, 80.30, 70.95, 15.25, 99.97, 85.56, 69.77
Is there an algorithm that can be efficiently implemented in PHP to find the outliers (if there are any) from an array of floats based on how far they deviate from the mean?

Answer 1

Ok let's assume you have your data points in an array like so:
<?php $dataset = array(20.50, 80.30, 70.95, 15.25, 99.97, 85.56, 69.77); ?>
Then you can use the following function (see comments for what is happening) to remove all numbers that fall outside of the mean +/- the standard deviation times a magnitude you set (defaults to 1):
<?php
function remove_outliers($dataset, $magnitude = 1) {
$count = count($dataset);
$mean = array_sum($dataset) / $count; // Calculate the mean
$deviation = sqrt(array_sum(array_map("sd_square", $dataset, array_fill(0, $count, $mean))) / $count) * $magnitude; // Calculate standard deviation and times by magnitude
return array_filter($dataset, function($x) use ($mean, $deviation) { return ($x <= $mean + $deviation && $x >= $mean - $deviation); }); // Return filtered array of values that lie within $mean +- $deviation.
}
function sd_square($x, $mean) {
return pow($x - $mean, 2);
}
?>
For your example this function returns the following with a magnitude of 1:
Array
(
[1] => 80.3
[2] => 70.95
[5] => 85.56
[6] => 69.77
)

Answer 2

For a normally distributed set of data, removes values more than 3 standard deviations from the mean.
<?php
function remove_outliers($array) {
if(count($array) == 0) {
return $array;
}
$ret = array();
$mean = array_sum($array)/count($array);
$stddev = stats_standard_deviation($array);
$outlier = 3 * $stddev;
foreach($array as $a) {
if(!abs($a - $mean) > $outlier) {
$ret[] = $a;
}
}
return $ret;
}

Answer 3

Topic: Detecting local, additive outliers in unordered arrays by walking a small window through the array and calculating the standard deviation for a certain range of values.
Good morning folks,
here is my solution much to late, but since I was looking for detecting outliers via PHP and could'nt find anything basic, I decided somehow smoothing a given dataset in a timeline of 24 h by simply moving a range of 5 items in a row through an unordered array and calculate the local standard deviation to detect the additive outliers.
The first function will simply calculate the average and deviation of a given array, where $col means the column with the values (sorry for the freegrades, this means that in an uncomplete dataset of 5 values you only have 4 freegrades - I don't know the exact english word for Freiheitsgrade):
function analytics_stat ($arr,$col,$freegrades = 0) {
// calculate average called mu
$mu = 0;
foreach ($arr as $row) {
$mu += $row[$col];
}
$mu = $mu / count($arr);
// calculate empiric standard deviation called sigma
$sigma = 0;
foreach ($arr as $row) {
$sigma += pow(($mu - $row[$col]),2);
}
$sigma = sqrt($sigma / (count($arr) - $freegrades));
return [$mu,$sigma];
}
Now its time for the core function, which will move through the given array and create a new array with the result. Margin means the factor to multiply the deviation with, since only one Sigma detects to many outliers, whereas more than 1.7 seems to high:
function analytics_detect_local_outliers ($arr,$col,$range,$margin = 1.0) {
$count = count($arr);
if ($count < $range) return false;
// the initial state of each value is NOT OUTLIER
$arr_result = [];
for ($i = 0;$i < $count;$i++) {
$arr_result[$i] = false;
}
$max = $count - $range + 1;
for ($i = 0;$i < $max;$i++) {
// calculate mu and sigma for current interval
// remember that 5 values will determine the divisor 4 for sigma
// since we only look at a part of the hole data set
$stat = analytics_stat(array_slice($arr,$i,$range),$col,1);
// a value in this interval counts, if it's found outside our defined sigma interval
$range_max = $i + $range;
for ($j = $i;$j < $range_max;$j++) {
if (abs($arr[$j][$col] - $stat[0]) > $margin * $stat[1]) {
$arr_result[$j] = true;
// this would be the place to add a counter to isolate
// real outliers from sudden steps in our data set
}
}
}
return $arr_result;
}
And finally comes the test function with random values in an array with length 24.
As for margin I was curious and choose the Golden Cut PHI = 1.618 ... since I really like this number and some Excel test results have led me to a margin of 1.7, above which outliers very rarelly were detected. The range of 5 is variable, but for me this was enough. So for every 5 values in a row there will be a calculation:
function test_outliers () {
// create 2 dimensional data array with items [hour,value]
$arr = [];
for ($i = 0;$i < 24;$i++) {
$arr[$i] = [$i,rand(0,500)];
}
// set parameter for detection algorithm
$result = [];
$col = 1;
$range = 5;
$margin = 1.618;
$result = analytics_detect_local_outliers ($arr,$col,$range,$margin);
// display results
echo "<p style='font-size:8pt;'>";
for ($i = 0;$i < 24;$i++) {
if ($result[$i]) echo "♦".$arr[$i][1]."♦ "; else echo $arr[$i][1]." ";
}
echo "</p>";
}
After 20 calls of the test function I got these results:
417 140 372 131 449 26 192 222 320 349 94 147 201 ♦342♦ 123 16 15
♦490♦ 78 190 ♦434♦ 27 3 276
379 440 198 135 22 461 208 376 286 ♦73♦ 331 358 341 14 112 190 110 266
350 232 265 ♦63♦ 90 94
228 ♦392♦ 130 134 170 ♦485♦ 17 463 13 326 47 439 430 151 268 172 342
445 477 ♦21♦ 421 440 219 95
88 121 292 255 ♦16♦ 223 244 109 127 231 370 16 93 379 218 87 ♦335♦ 150
84 181 25 280 15 406
85 252 310 122 188 302 ♦13♦ 439 254 414 423 216 456 321 85 61 215 7
297 337 204 210 106 149
345 411 308 360 308 346 ♦451♦ ♦77♦ 16 498 331 160 142 102 ♦496♦ 220
107 143 ♦241♦ 113 82 355 114 452
490 222 412 94 2 ♦480♦ 181 149 41 110 220 ♦477♦ 278 349 73 186 135 181
♦39♦ 136 284 340 165 438
147 311 246 449 396 328 330 280 453 374 214 289 489 185 445 86 426 246
319 ♦30♦ 436 290 384 232
442 302 ♦436♦ 50 114 15 21 93 ♦376♦ 416 439 ♦222♦ 398 237 234 44 102
464 204 421 161 330 396 461
498 320 105 22 281 168 381 216 435 360 19 ♦402♦ 131 128 66 187 291 459
319 433 86 84 325 247
440 491 381 491 ♦22♦ 412 33 273 256 331 79 452 314 485 66 138 116 356
290 190 336 178 298 218
394 439 387 ♦80♦ 463 369 ♦104♦ 388 465 455 ♦246♦ 499 70 431 360 ♦22♦
203 280 241 319 ♦34♦ 238 439 497
485 289 249 ♦416♦ 228 166 217 186 184 ♦356♦ 142 166 26 91 70 ♦466♦ 177
357 298 443 307 387 373 209
338 166 90 122 442 429 499 293 ♦41♦ 159 395 79 307 91 325 91 162 211
85 189 278 251 224 481
77 196 37 326 230 281 ♦73♦ 334 159 490 127 365 37 57 246 26 285 468
228 181 74 ♦455♦ 119 435
328 3 216 149 217 348 65 433 164 473 465 145 341 112 462 396 168 251
351 43 320 123 181 198
216 213 249 219 ♦29♦ 255 100 216 181 233 33 47 344 383 ♦94♦ 323 440
187 79 403 139 382 37 395
366 450 263 160 290 ♦126♦ 304 307 335 396 458 195 171 493 270 434 222
401 38 383 158 355 311 150
402 339 382 97 125 88 300 332 250 ♦86♦ 362 214 448 67 114 ♦354♦ 140 16
♦354♦ 109 0 168 127 89
450 5 232 155 159 264 214 ♦416♦ 51 429 372 230 298 232 251 207 ♦322♦
160 148 206 293 446 111 338
I hope, this will help anyone in the present or future.
Greetings
P.S. To further improve this algorithm you may add a counter, which makes sure, that a certain value must for instance be found at least 2 times, that means in 2 different intervals or windows, before it is labeled as outlier. So a sudden jump of the following values does not make the first value the villain. Let me give you an example:
In 3,6,5,9,37,40,42,51,98,39,33,45 there is an obvious step from 9 to 37 and an isolated value 98. I would like to detect 98, but not 9 or 37.
The first interval 3,6,5,9,37 would detect 37, the second interval 6,5,9,37,40 not. So we would not detect 37, since there is only one problematic interval or one match. Now it should be clear, that 98 counts in 5 intervals and is therefore an outlier. So lets declare a value an outlier, if it "counts" at least 2 times.
Like so often we have to look closely the borders, since they have only one interval, and make for these values an exception.