Find double files using PHP with high performance - php

Question

I've got around 25000 files scattered around many folders which vary between 5MB and 200MB on 2 external hard drives. I need to find out which of these are duplicate, leaving only the unique files on the drives.
Currently im doing md5_file() over each source file and compare these to see if the same file has been found before. The issue with that is, md5_file() could easily take more than 10 seconds to execute and I've seen it even taking up to a minute for some files. If I let this script run in it's current form, that would mean this process will take more than a week to finish.
Note that I'm saving each hash after one has been made, so I dont have to re-hash each file on each run. Thing is that all these files are yet to be hashed.
I'm wondering what I could do to speed this up. I need to finish this in less than 5 days, so a script that takes more than a week is no option. I was thinking multithreading (using pthread) could be a solution, but as the drives are so slow and my CPU is not the issue, I don't think this would help. What else is there I could do?

Answer 1

As you guessed, it's hard to tell if you can see any gains by using threading ...
However, I decided I would write a nice pthreads example based on your idea, I think it illustrates well things you should do while threading ...
Your mileage will vary, but here's the example all the same:
<?php
/* create a mutex for readable logging output */
define ("LOG", Mutex::create());
/* log a message to stdout, use as thread safe printf */
function out($message, $format = null) {
$format = func_get_args();
if ($format) {
$message = array_shift(
$format);
Mutex::lock(LOG);
echo vsprintf(
$message, $format
);
Mutex::unlock(LOG);
}
}
/*
Sums is a collection of sum => file shared among workers
*/
class Sums extends Stackable {
public function run(){}
}
/* Worker to execute sum tasks */
class CheckWorker extends Worker {
public function run() {}
}
/*
The simplest version of a job that calculates the checksum of a file
*/
class Check extends Stackable {
/* all properties are public */
public $file;
public $sum;
/* accept a file and Sums collection */
public function __construct($file, Sums &$sums) {
$this->file = $file;
$this->sums = $sums;
}
public function run(){
out(
"checking: %s\n", $this->file);
/* calculate checksum */
$sum = md5_file($this->file);
/* check for sum in list */
if (isset($this->sums[$sum])) {
/* deal with duplicate */
out(
"duplicate file found: %s, duplicate of %s\n", $this->file, $this->sums[$sum]);
} else {
/* set sum in shared list */
$this->sums[$sum] = $this->file;
/* output some info ... */
out(
"unique file found: %s, sum (%s)\n", $this->file, $sum);
}
}
}
/* start a timer */
$start = microtime(true);
/* checksum collection, shared across all threads */
$sums = new Sums();
/* create a suitable amount of worker threads */
$workers = array();
$checks = array();
$worker = 0;
/* how many worker threads you have depends on your hardware */
while (count($workers) < 16) {
$workers[$worker] = new CheckWorker();
$workers[$worker]->start();
$worker++;
}
/* scan path given on command line for files */
foreach (scandir($argv[1]) as $id => $path) {
/* #TODO(u) write code to recursively scan a path */
$path = sprintf(
"%s/%s",
$argv[1], $path
);
/* create a job to calculate the checksum of a file */
if (!is_dir($path)) {
$checks[$id] = new Check(
$path, $sums);
/* #TODO(u) write code to stack to an appropriate worker */
$workers[array_rand($workers)]->stack($checks[$id]);
}
}
/* join threads */
foreach ($workers as $worker) {
$worker->shutdown();
}
/* output some info */
out("complete in %.3f seconds\n", microtime(true)-$start);
/* destroy logging mutex */
Mutex::destroy(LOG);
?>
Play around with it, see how different numbers of workers affects runtime, and implement your own logic to delete files and scan directories (this is basic stuff you should know already, left out to make for a simple example) ...

Answer 2

You could try to find possible duplicates by only looking at the file size. Then only if multiple files have the same size you need to hash them. This is probably faster, since looking up files sizes is not much of an effort.