Random code execution stops when using Eloquent chunk() - php

Question

I have a method for a scheduled system cleanup that goes through all the files in the "storage" table, selecting the type of files we need (property photos), and then going through each of them defining if corresponding listing still exists in the database. If not, removing the record from the DB and removing the file itself.
Now about the problem. Originally I didn't use chunk(), it was just Model::all() to select everything and it all worked well. But at this point I've got 200000 records in that storage table and these operations began to crash because of enormous memory consumption. So I decided to go with chunk().
So the problem is that now it works as it should, however, at some random moments (somewhere around the middle of the process) the code execution just stops as if the operation was completed, so no errors logged anywhere and the task is not fully completed.
Can you please suggest what can be the cause of such strange behavior?
public function verifyPhotos() {
// Instantiating required models and putting them into a single array so they can be passed to a closure
$models = [];
$models['storage'] = App::make('Store');
$models['condo'] = App::make('Condo');
$models['commercial'] = App::make('Commercial');
$models['residential'] = App::make('Residential');
// Obtaining and processing all records from the storage chunk by chunk
$models['storage']->where('subject_type', '=', 'property_photo')->chunk(10000, function($files) use(&$models) {
// Going through each record in current chunk
foreach ($files as $photo) {
// If record's subject type is Condo
if ($photo->subject_name == 'CondoProperty') {
// Selecting Condo model to work with
$current_model = $models['condo'];
// If record's subject type is Commercial
} elseif ($photo->subject_name == 'CommercialProperty') {
// Selecting Commercial model to work with
$current_model = $models['commercial'];
// If record's subject type is Residential
} elseif($photo->subject_name == 'ResidentialProperty') {
// Selecting Residential model to work with
$current_model = $models['residential'];
}
// If linked listing doesn't exist anymore
if (!$current_model->where('ml_num', '=', $photo->owner_id)->first()) {
// Deleting a storage record and physical file
Storage::delete('/files/property/photos/'.$photo->file_name);
$models['storage']->unregisterFile($photo->id);
}
}
});
}

Answer 1

Using chunk() in Eloquent will add a limit and an offset to your SQL query and execute it for every chunk. If you change your data in the database reducing the rows matched by the query, you will skip over the matching rows in the next execution because of the offset.
I.e. if you have 9 rows with id = 1...9 and subject_type = 'property_photo' and you use chunk(3, ...) the resulting queries are:
select * from store where subject_type = 'property_photo' limit 3 offset 0;
select * from store where subject_type = 'property_photo' limit 3 offset 3;
select * from store where subject_type = 'property_photo' limit 3 offset 6;
select * from store where subject_type = 'property_photo' limit 3 offset 9;
If you inside the each chunk set subject_type = 'something' for each row, those rows no longer match and the next query which offsets by 3 would effectively skip the next 3 matching.
It may be possible to use the Collection::each() closure instead like below, although it would still have to load the entire result set into a collection:
$models['storage']->where('subject_type', '=', 'property_photo')->get()->each(function ($photo) use (&$models) {
if ($photo->subject_name == 'CondoProperty') {
//...
}
//...
});
Remember you can also run DB::disableQueryLog(); to save memory on large database operations.

Answer 2

You should add try ... catch to some suspicious codes and print exception message into log file. I also once found the same problem and eventually found that it was also about memory consumption.
Most suspicious part for me is reusing models, $current_model->where(). I suspect that the memory may not be released after each query. Basically each query should be used only once. Is there any reason to reuse it?
Try change to $current_model = App::make('YourModel'); instead of reusing via $models and see if it solves.