Most efficient way of implementing an email queue in Laravel - php

I want to implement a queue for sending out emails in Laravel. I have the queue working fine, but am worried about efficiency. These are my settings:
I have created the jobs table and set up the .env file, to use the queues with my local database.
I have set up this crontab on the server:
* * * * * php /var/www/imagine.dev/artisan schedule:run >> /dev/null 2>&1
And have set up a schedule in app\Conosle\Kernel.php, so I dont have to manually enter the 'queue:listen' every time through console.
$schedule->command('queue:listen');
Now to my question. I would like to know if this is efficient? I am worried about having the queue:listen running all the time in the background consuming cpu and memory.
I have been trying to only run the queue:listen once every 5 minutes, and then put it to sleep with
$schedule->command('queue:listen --sleep 300');
but again, am not sure if this is the best approach.
Another thing I tried is using 'queue:work', but this only processes one queue at a time.
Ideally, I would like a way, to process all the queues every 5 minutes, avoiding a constant use of memory and cpu.
What is the best approach?

Not sure which version of Laravel you're using, but I suspect it's 5.2 or earlier.
You do not need to run this every minute, it continues to run until it's manually stopped.
From Laravel 5.2 documentation:
Note that once this task has started, it will continue to run until it is manually stopped. You may use a process monitor such as Supervisor to ensure that the queue listener does not stop running.
So maybe you want to look into Supervisor
Also, if this is helpful at all, you can chain onto $schedule, ->everyFiveMinutes(). There are several other methods available as well. Laravel Scheduling

Related

Laravel: Schedule a job or an artisan command?

I have a Product model with id,name,price.
The price value is stored in an external API and i need to fetch it every minute in order to update it in the database.
Looking through the Laravel documentation I found two ways to implement:
Create an artisan command (https://laravel.com/docs/8.x/artisan) and add it to task scheduling (https://laravel.com/docs/8.x/scheduling#scheduling-artisan-commands)
Create a job (https://laravel.com/docs/8.x/queues) and add it to task scheduling (https://laravel.com/docs/8.x/scheduling#scheduling-artisan-commands)
First of all, is there any other approach i should take in consideration?
If not, which one of the above would be the best approach and why is it correct for my use case?
As per my comments on one of your previous questions on this topic, whether you use a queue or not depends on your use case.
An Artisan command is a process that executes once and performs a task or tasks and then exits when that task is complete. It is generally run from the command line rather than through a user action. You can then use the task scheduling of your command's host operating system (e.g. a CRON job) to execute that command periodically. It will faithfully execute it when you schedule it to be done.
A Queued job will execute when the Job turns up next in the queue, in priority order. Let's say you send your API call (from your other post) to the queue to be processed. Another system then decides it needs to send out emails urgently (with a higher priority). Suddenly, your Job, which was next, is now waiting for 2000 other Jobs to finish (which might take a half hour). Then, you're no longer receiving new data until your Job executes.
With a scheduled job, you have a time critical system in place. With queues, you have a "when I get to it" approach.
Hope this makes the difference clearer.
With laravel it is a lot easy to use the built in scheduler. You have to add only one entry to the crontab and that is to run the command php artisan schedule:run EVERY MINUTE on your project. After that you dont have to thing about configuring the crontab on the server, you just add commands to the laravel scheduler and they will work as expected.
You should probably use Cron Job Task Scheduling which would be the first approach you mentioned.
Commonly for this type of use-cases commands are the easiest and cleanest approach.
There are a few things to do in order to make it work as expected:
Create a new command that will need to take care of hitting the endpoint and storing the retrieved data to the database
In Kernel.php file register your command and the frequency of running (each minute)
Run php artisan schedule:run
You can read more about how to create it here:

Cron job auto retry if any job get failed

I have a cron job that run every 5 Hours. It calls a PHP script , this script will do a call to an external API to sync some data.
The problem is sometimes I'm getting timeout from the API and the job will fail.
Are there any mechanisms to let cron tab do auto retry or auto recover the jobs that are failed?
I have tried to do an extra job and call it in case of any failures manually.
What is the best approach to do so?
Cron does only run once at specific time or every minutes/hours/days etc. It doesn't check the return code. So it's not that easy peasy lemon squeezy at all...
In my opinion you have a few options how to do it:
Create a some kind of scheduler where you can write your CRON job again if it fails, in this case you will need one more CRON job to read you scheduler and run proper command. Scheduler can be database / file / NoSQL based. In scheduler you can have flag like (bool) executed which will let scheduler know which tasks are already done.
Use queues (f.ex. Rabbit) to call it self again when fail.
Use framework, I'm using Symfony to manage own created commands to execute them (check second link below) based on database, using also enqueue/enqueue-bundle package to manage queues in Symfony.
I think if you are not so advanced with PHP I'd recommend to go for self made scheduler based on database (MySQL / PostgreSQL / NoSQL) with Symfony (check second link below). In this case you just have to SELECT all non executed record (commands) from database and just run them.
Lecture
Laravel - Queues, retrying failed jobs
Symfony - calling another commands in command
Queues package for PHP (incl. Symfony)
enqueue/enqueue-bundle
What you can do is something like this:
https://crontab.guru/#1_0-23_13__
1 0-23 13 * *
Start the job past 1 minute at every hour on the 13th of each month.
“At minute 1 past every hour from 0 through 23 on day-of-month 13.”
...then in your code you'd have some logic to detect if the process\script already ran correclty... if yes, skip the run attempt; otherwise let it run and then have a flag set to check against on the subsequent attempt run.
Hope you get the idea.
You can use supervisord :
supervisord website
Or handle API timeout in code.

How to manually release all the queued jobs in Laravel?

I need to know if there is a way to use the internal laravel api to force the release of all queued jobs. The reason is that we have a queue implementation and there we have a mechanism that releases the execution of a job 5 minutes, if there was a problem during the job execution. The problem is that is required to have some sort of refresh feature that triggers all of those "delayed" jobs manually, since we need a bit of control of when to run those delayed jobs, keeping the fail-safe mechanism intact. There is some way to implement this using Laravel??
You can run the php artisan queue:work commando to start the Queue Work. If you wish start this from the code, you can call this command programmatically

Limit to cron job with PHP

I have a Cron Job with PHP which I want to set up on my webhost, but at the moment the script takes about 20 seconds to run with only 3 users data being refreshed. If I get a 1000 users - gonna take ages. Is there an alternative to Cron Job? Will my web host let me run a cron job which takes, for example, 10 minutes to run?
Your cron job can be as long as you want.
The main problem for you is that you must ensure the next cron job execution is not occuring while the first one is still running. You have a lot of solutions to avoid it, basically use a semaphore.
It can be a lock file, a record in database. Your cron job should check if the previous one is finished or not. A good thing is maybe sending you an email if he cannot run because of a long previous job (this way you'll have some notice alerting you that something is maybe getting wrong) By default cron jobs with bad error dstatus on exit are outputing all the standard output to the email of the account running the job, depending on how is configured the platform you could use this behavior or build an smtp connexion on the job (or store the alert in a database table).
If you want some alternatives to cron jobs you should have a look at work queues. You can mix work queues with a cron job, or use work queue in apache-php envirronment, lot of solutions, but the main idea is to make on single queue of things that should be done, and execute them one after the other (but be careful, if you handle theses tasks very slowly you'll get a big fat waiting queue).
A cron job shouldn't have any bearing on how long it's 'job' takes to complete. If you're jobs are taking 20 seconds to complete, it's PHP's fault, not cronjob.
Will my web host let me run a cron job which takes, for example, 10 minutes to run?
Ask your webhost.
If you want to learn about optimizing php scripts, take a look at Profiling PHP Code.

cron jobs or PHP scheduler

I am using MYSQL as my database and PHP as my programming language.I wanted to run a cron job which would run until the current system date matches the "deadline(date)" column in my database table called "PROJECT".Once the dates are same an update query has to run which would change the status(field of project table) from "open" to "close".
I am not really sure if cron jobs are the best way or I could use triggers or may be something else.Also I am using Apache as my web server and my OS is windows vista.
Also which is the best way to do it? PHP scheduler or cron jobs or any other method? can anybody enlighten me?
I think your concept needs to change.
PHP cannot schedule a job, neither can MySQL. Triggers in MySQL execute when a mysql query occurs, not at a specific time. Neither
This limitation usually isn't a problem in web development. The reason is because your PHP application should control all data going in and out. Usually, this means just the HTML that displays that data, or other formats to users, or other programs.
In your case you can think about it this way. The deadline is a set date. You can treat it as data, and save it to your database. When the deadline occurs is not important, it is that the data you have sent in your database is viewed correctly.
When a request is made to your application, check if the date of the deadline is in the past, if it is, then display that the project is closed - or update that the project is closed, just before display.
There really is no reason to update data independantly of your PHP application.
Usually, the only things you want to schedule are jobs that would affect your application in terms of load, or that need to be done only once, or where concurrency or time is an issue.
In your case none of those apply.
PS: I haven't tried PHPscheduler but I can guess it isn't a true scheduler. Cron is a deamon that sleeps until a given task is due in its queue, executes the task, then sleeps till the next one is due (at least thats what it does in the current algorithm). PHP cannot do that without the sockets and fork extensions, as special setup. So PHPscheduler is most likely just checking if a date for a task has expired, on each load of a webpage (whenever PHP executes a page). This is no different then you just checking if the date on the project has expired, without the overhead of PHPScheduler.
I would always go for a cron job for anything scheduling related.
The big bonus point is that you can echo info out as well and it get's emailed to you.
You'll find once you start using cronjobs, it's hard to stop.
cron does not exist, per se, in vista, but what does exist is the standard windows scheduling manager which you can run with a command line like "php -q -f myfile.php" which will execute the php file at the given time.
you can also use a port of the cron program, there are many out there.
if it is not critical to the second, any windows scheduling application will do, just be sure to have you PHP bin path in your PATH variable for simplicity.
For Windows CRON jobs I cannot recommend PyCron enough.
While CRON and Windows Scheduled Tasks are the tried and true ways of scheduling jobs/tasks to run on a regular basis, there are use cases where having a different scheduled task in CRON/Windows can become tedious. Namely when you want to let users schedule things to run, or for instances where you prefer simplicity/maintainability/portability/etc or all of the above.
In cases where I prefer to not use CRON/Windows for scheduled tasks, I build into the application a task scheduling system. This still requires 1 CRON job or Windows Task to be scheduled. The idea is to store Job details in the database (job name, job properties, last run time, run interval, anything else that is important for your implementation). You then schedule a "Master" job in CRON or Windows which handles running all of your other jobs for you. You'll need this master job to run at least as often as your shortest interval; if you want to be able to schedule jobs that run every minute the master job needs to run every minute.
You can then launch each scheduled job in the background from PHP with minimal effort (if you want). In memory constrained systems you can monitor memory usage or keep track of the PIDs (various methods) and limit to N jobs running at a given time.
I've had a great deal of success with this method, YMMV however based on your needs and your implementation.
how about PHPscheduler..R they not better than cronjobs? I think crons would be independent of the application hence would be difficult if one has to change the host..i am not really sure though..It would be great if anyone can comment on this!! Thanks!

Categories