How to handle mass mail sending with queue timeout - php

Question

I have table with 100K+ emails where I want to daily send some email:
I added schedule in app\Console\Kernel.php:
$schedule->job(new SendDailyEmails)->dailyAt('09:00');
Inside Job I have:
$users = User::all();
foreach($users as $user){
Maill:to($user->email)->send(new DailyMail($user));
$status = 'sent';
if( in_array($user->email, Mail::failures()) ){
$status = 'failed';
Log::error($user->email . ' was not sent.');
}else{
Log::info($user->email . ' was sent.');
}
SentMail::create([
'email' => $user->email,
'status' => $status
]);
}
This works fine, but after some time this stops probably because job timeout. In failed_jobs table I get MaxAttemptsExceededException with message that Job attempted too many times or run too long. Since I set queue tries max to 3 within supervisor it should and do go 3 times only. And by testing things it did not try to attempt again because I got one mail instead of 3.
So it comes to timeout and I am not sure what is default value but does it matter since I will not know how much time it would take to send all emails?
Should I divide mails into groups of 50 and call separate job instances for each group?
Anyone have a good working answer for this?

Answer 1

If you have a look at the official documentation you will find out that every single mail can be queued.
So, you should change your job from
Mail:to($user->email)->send(new DailyMail($user));
to
Mail:to($user->email)->queue(new DailyMail($user));
In this way you will push into the queue every single mail your job generates. I suggest you to create a specific queue and use system like laravel horizon for a better monitoring.
Also remember to chunk and delay the sending because your application can run a timeout error, but also providers like mailgun may block your sending if they see an unusual activity.

Answer 2

Instead of trying to send 100k+ emails all at once in a class, dispatch 100k+ instances of the Job to a queue worker
$users = User::all();
foreach($users as $user){
$schedule->job(new SendDailyEmails($user))->dailyAt('09:00');
}
Now Laravel will Stack a 100k+ jobs in a queue and attempt to send one email per one user at a time
class SendDailyEmails implements ShouldQueue
{
public $user;
public function __construct(User $user)
{
$this->user = $user;
}
Maill:to($this->user->email)->send(new DailyMail($user));
$status = 'sent';
if( in_array($this->user->email, Mail::failures()) ){
$status = 'failed';
Log::error($this->user->email . ' was not sent.');
}else{
Log::info($this->user->email . ' was sent.');
}
SentMail::create([
'email' => $this->user->email,
'status' => $status
]);