Check if image exists - database or file lookup? - php

Question

I hash a file's name based on its contents and store this name reference in a database, and store the file on a server.
Would it be more efficient (quicker) to check for duplicate files (and therefore not re-upload) via checking its name in a database or by checking if the file exists on the server?
There would be 1000s of files.

Answer 1

I had the same issue, we have roughly 40k images and the duplicates were loading heavy on our server, especially with image license management, as the same license had to be added to the same image multiple times.
I recommmend a database lookup. It's much faster as your collection of files grows. a 40k table scan takes something like 20 milliseconds. A 40k file search on disk runs in a few seconds, which gets annoying fast.
To solve this we changed how images were uploaded, so we don't get duplicate files, but multiple database records that reference the same physical file on disk. This gives us speed for looking up the file data, without having the "file" or even knowing where the actual file is.
We also don't store the file as the original filename, but as hexadicimal hash based on date and time, so we don't get conflicting filenames, and have no delivery issues due to special characters, spaces, etc... and store the original file name in a database field for lookup purposes.
Our images have their "metadata" stored in the database, with a hexadecimal file name and an "original" filename. It's really fast to check against this database, and then retrieve the file link when there's a match/relation. This also allows checking if this file has already been uploaded, as you don't need to scan the entire directory structure with alt the images, which can take up a significant amount of time.
This is the code I use, you can use something similar. Note that this is using laravel's eloquent, but it's fairly easy to replicate in pure mysql.
First you get an instance to query the file model table.
Then you check for a file where the original filename, file size, content type and other meta data that shouldn't change are the same.
If they are the same, make your file entry a duplicate of the original entry(In my case this allows modifying image titles and descriptions for each reference)
$file = new $FILE();
$existingFile = $file->newQuery()->where('file_name',$uploadedFile->getClientOriginalName())
->where('file_size',$uploadedFile->getSize())
->where('content_type',$uploadedFile->getMimeType())
->where('is_public',$fileRelation->isPublic())->limit(1)->get()->first();
if($existingFile) {
$file->disk_name = $existingFile->disk_name;
$file->file_size = $existingFile->file_size;
$file->file_name = $existingFile->file_name;
$file->content_type = $existingFile->content_type;
$file->is_public = $existingFile->is_public;
}
else {
$file->data = $uploadedFile;
$file->is_public = $fileRelation->isPublic();
}
then as the file is deleted, you need to check if it's the "last one"
public function afterDelete()
{
try {
$count = $this->newQuery()->where('disk_name','=',$this->disk_name)
->where('file_name','=',$this->file_name)
->where('file_size','=', $this->file_size)
->count();
if(!$count) {
$this->deleteThumbs();
$this->deleteFile();
}
}
catch (Exception $ex) {
traceLog($ex->getMessage() . '\n' . $ex->getTraceAsString());
}
}