Phalcon Query super slow inside large loop - php

Question

I'm using Phalcon 3.0.4. I made a foreach on each file inside my folder. Currently I have just 4000 files. I did a findFirst to check if the filename already exist in MySQL (I have 100 000 rows in my table). But when I use findFirst, the response is super slow (I have to wait 20 minutes to get a response). Here is my code :
$dir = new FilesystemIterator("files/path/to/my/files/");
foreach ($dir as $file) {
if ($file->getExtension() == 'json') {
$filename = $file->getFilename();
$explode_filename = explode("_", $filename);
$date = $explode_filename[0];
$unformatted_date = DateTime::createFromFormat("Ymd-His", $date);
$date_server = $unformatted_date->format("Y-m-d H:i:s");
$timestamp_app = $explode_filename[2];
$date_app = date("Y-m-d H:i:s", $timestamp_app/1000);
echo $date_server;
$json_data = json_decode(file_get_contents($file), true);
$scan = Scans::findFirst(array(
"name = :name:",
"bind" => array("name" => $filename)
));
if (!$scan) {
...
}
}
}
I tried to make my query with the QueryBuilder PHQL but I have the same result:
$scan = $this->modelsManager->createBuilder()
->from("Scans")
->where("name = :name:", ["name" => $filename])
->limit(1)
->getQuery()
->execute();
If I remove the findFirst or queryBuilder the response is ~30ms but with the findFirst it will takes ~20 minutes... How can I do to increase the performance of the search in my table ?

Answer 1

By changing your code to better performing one:
$dir = new FilesystemIterator("files/path/to/my/files/");
$fileNames = [];
foreach ($dir as $file) {
if ($file->getExtension() == 'json') {
$filename = $file->getFilename();
$explode_filename = explode("_", $filename);
$date = $explode_filename[0];
$unformatted_date = DateTime::createFromFormat("Ymd-His", $date);
$date_server = $unformatted_date->format("Y-m-d H:i:s");
$timestamp_app = $explode_filename[2];
$date_app = date("Y-m-d H:i:s", $timestamp_app/1000);
echo $date_server;
$json_data = json_decode(file_get_contents($file), true);
// save the above data to some arrays
$fileNames[] = $fileName;
}
}
$scans = Scans::find([
'columns' => 'check only columns you need, otherwise you will have full models with hydration',
'conditions' => 'name IN ({fileNames:array})',
'group' => 'name',
'bind' => [
'fileNames' => $fileNames
]
]);
foreach($fileNames as $fileName) {
$filteredScans = $scans->filter(function($scan) use ($fileName) {
return $scan->name == $fileName;
}
if(!$filteredScans) {
// do here whatever
}
}
This solution can be memory heavy though, then you could include here some paginations like do some limit like proper for and do 100-10000 rows at once depending how much RAM you have.

Answer 2

create index on Scans.name
use group by Scans.name (if not uniq)
set some columns then be use