php - Remove duplicates from current array while looping through the same array? - php

Question

i'm playing with cURL to crawl pages and extract links. This is some code that targets my issue.
for($i = 0; $i < sizeof($links); $i++){
$response = crawl($links[$i]);
//inside this loop i extract links for each crawled html
$newLinks = getLinks($response);
//I need to append these new links to current array in loop
$links= array_values(array_unique(array_merge($links, $newLinks));
}
I need to prevent for duplicate links so i don't crawl twice. I wonder if this is a safe approach or if it's right at all, since array_values whould reindex the elements of the array and while in loop the crawling could run twice for some link.
I could test with in_array() against $links and $newLinks to avoid duplicates but i wonder what happens when doing like my sample here.

Answer 1

This example work if you get only one link by function getLinks();
$newLink = getLinks($response);
if(!in_array($newLink,$links)){
$links=array_merge($links, array($newLink));
}
If you get more link, you can use foreach for links
foreach($newLinks as $newLink){
if(!in_array($newLink,$links)){
$links=array_merge($links, array($newLink));
}
}

Answer 2

TL/DR: Use array_merge($links, array_diff($newLinks, $links));
My justification:
Welcome to the land of exponential growth. As your collected links array grows, the time it takes will go through the roof. Give array_merge(array_diff($links, $new_links)) a chance instead. Using this benchmarking code:
function get($num)
{
$links = array();
for($i=0;$i<rand(5,20);$i++)
{
$links[] = rand(1,$num);
}
return $links;
}
function test($iter, $num_total_links)
{
$unique_time = 0;
$unique_links = array();
$diff_time = 0;
$diff_links = array();
for($i=0;$i<$iter;$i++)
{
$new_links = get($num_total_links);
$start = microtime(true);
$unique_links =
array_values(array_unique(array_merge($unique_links, $new_links)));
$end = microtime(true);
$unique_time += $end - $start;
$start = microtime(true);
$diff_links = array_values(array_merge(array_diff($new_links, $diff_links)));
$end = microtime(true);
$diff_time += $end - $start;
}
echo $unique_time . ' - ' . $diff_time;
}
You can tweak the values; if you expect to surf a large number of pages with relatively few links in common, pick a large (not too large or it'll take forever) $iter and $num_total_links. If you're likely to see many of the same link, reduce the $num_total_links accordingly.
What it boils down to is that a merge and then unique operation always requires you to merge in the same number of links, while a diff and then merge only requires you to merge in the links you want to add. This is nearly always more efficient, even at numbers that aren't exactly huge; surfing 500 pages with between 5 and 20 links makes a huge difference in time, and even a small number of pages can show a marked difference.
Looking at the data, foreach isn't a bad way to go as compared with array_unique; doing a similar benchmark, foreach consistently beat array_unique. But both are O(n^2), with foreach just growing more slowly. array_diff preserves O(n) time, which is your best case; your algorithm is never going to get any faster than some multiple of the number of pages you visit. PHP's built-in array functions are going to be faster than pretty much any solution written in PHP.
Here is the data, with the random factors taken out because all they did was cause deviations, not affect the results. I've also ramped up the possible "universe" of URLs to a large number for illustration purposes; at a smaller number, both array_unique and foreach produce graphs which look almost linear, albeit that they continue to be outperformed by array_diff. If you plug this into Google Charts, you can see just what I'm talking about.
['Number of Pages', 'array_unique', 'array_diff', 'foreach'],
[1, 6.9856643676758E-5, 6.1988830566406E-5, 0.00012898445129395],
[11, 0.0028481483459473, 0.00087666511535645, 0.0014169216156006],
[21, 0.0091345310211182, 0.0017409324645996, 0.0029785633087158],
[31, 0.019546031951904, 0.0023491382598877, 0.0046005249023438],
[41, 0.036402702331543, 0.0032360553741455, 0.006026029586792],
[51, 0.055278301239014, 0.0039372444152832, 0.0078754425048828],
[61, 0.082642316818237, 0.0048537254333496, 0.010209321975708],
[71, 0.11405396461487, 0.0054631233215332, 0.012364625930786],
[81, 0.15123820304871, 0.0062053203582764, 0.014509916305542],
[91, 0.19236493110657, 0.007127046585083, 0.017033576965332],
[101, 0.24052715301514, 0.0080602169036865, 0.01974892616272],
[111, 0.29827189445496, 0.0085773468017578, 0.023083209991455],
[121, 0.35718178749084, 0.0094895362854004, 0.025837421417236],
[131, 0.42515468597412, 0.010404586791992, 0.029412984848022],
[141, 0.49908661842346, 0.011186361312866, 0.033211469650269],
[151, 0.56992983818054, 0.011844635009766, 0.036608695983887],
[161, 0.65314698219299, 0.012562274932861, 0.039996147155762],
[171, 0.74602556228638, 0.013403177261353, 0.04484486579895],
[181, 0.84450364112854, 0.014075994491577, 0.04839038848877],
[191, 0.94431185722351, 0.01488733291626, 0.052026748657227],
[201, 1.0460951328278, 0.015958786010742, 0.056291818618774],
[211, 1.2530679702759, 0.016806602478027, 0.060890197753906],
[221, 1.2901678085327, 0.017560005187988, 0.065101146697998],
[231, 1.4267380237579, 0.018605709075928, 0.070043087005615],
[241, 1.5581474304199, 0.018914222717285, 0.075717210769653],
[251, 1.8255474567413, 0.020106792449951, 0.08226203918457],
[261, 1.8533885478973, 0.020873308181763, 0.085562705993652],
[271, 1.999392747879, 0.021762609481812, 0.15557670593262],
[281, 2.1670596599579, 0.022242784500122, 0.098419427871704],
[291, 2.4296963214874, 0.023237705230713, 0.10490798950195],
[301, 3.0475504398346, 0.031109094619751, 0.13519287109375],
[311, 3.0027780532837, 0.02937388420105, 0.13496232032776],
[321, 2.9123396873474, 0.025942325592041, 0.12607669830322],
[331, 3.0720682144165, 0.026587963104248, 0.13313007354736],
[341, 3.3559355735779, 0.028125047683716, 0.14407730102539],
[351, 3.5787575244904, 0.031508207321167, 0.15093517303467],
[361, 3.6996841430664, 0.028955698013306, 0.15273785591125],
[371, 3.9983749389648, 0.02990198135376, 0.16448092460632],
[381, 4.1213915348053, 0.030521154403687, 0.16835069656372],
[391, 4.3574469089508, 0.031461238861084, 0.17818260192871],
[401, 4.7959914207458, 0.032914161682129, 0.19097280502319],
[411, 4.9738960266113, 0.033754825592041, 0.19744348526001],
[421, 5.3298072814941, 0.035082101821899, 0.2117555141449],
[431, 5.5753719806671, 0.035769462585449, 0.21576929092407],
[441, 5.7648482322693, 0.035907506942749, 0.2213134765625],
[451, 5.9595069885254, 0.036591529846191, 0.2318480014801],
[461, 6.4193341732025, 0.037969827651978, 0.24672293663025],
[471, 6.7780020236969, 0.039541244506836, 0.25563311576843],
[481, 7.0454154014587, 0.039729595184326, 0.26160192489624],
[491, 7.450076341629, 0.040610551834106, 0.27283143997192]