i have my Laravel project setup on Ubuntu Server where emails are being sent using Jobs.
Below is my laravel-worker file in /etc/supervisor/laravel-worker.conf
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /home/forge/app.com/artisan queue:work sqs --sleep=3 --tries=3
autostart=true
autorestart=true
user=forge
numprocs=8
redirect_stderr=true
stdout_logfile=/home/forge/app.com/worker.log
stopwaitsecs=3600
My jobs are being dispatched in Controller as this
dispatch(new UserRegistrationEmail($data, $token));
in env file i have
QUEUE_CONNECTION=database
This was all working fine and emails were being sent in background but for some reason emails stopped going automatically and queues are not being attempted . My Jobs table
if i am logged in through Putty and run command php artisan queue:work it executes the jobs and emails are sent, Now until dont ctrl+c this command it keeps processing upcoming job requests as well , as seen in screenshot .
I have cleared config cache, restarted queue, cleared cache but nothing helped. Can someone please help with this , This is quite frustrating
When you dispatch a job through the code (like App\Jobs\YourJob::dispatch(); ) It will create a record in the jobs table, with all the information needed to run this job.
Once you run php artisan queue:work it starting to fetch these jobs from the database, as a queue (the latest one first) and run the commands.
In order to monitor and fetch and run these jobs when they are presented in the database I use the command php artisan queue:work --stop-when-empty and run this command every minute in my webserver as a cron job.
In this case everyminute it checks the datebase and if there is any job to handle it does.
Related
I want to fetch the data from the server at the same time by using multithreading.
The reason behind multi-threading is to avoid load on the server and its resources.
So, I found laravel queues and since I am new in laravel I don't know much about it but after r&d I did the work and develop a job that dispatches queues one after another BUT I want that queue will start at the same time
You can install and use Supervisor, Then you can config it by numprocs=8 property to run e.g. 8 queue processes.
Here is the Laravel documentation about installing and configuring Supervisor: https://laravel.com/docs/master/queues#supervisor-configuration
You can also categorize your queues and dispatch your job to a particular queue (here is the docs) then use Supervisor with a config file for each queue.
Installing Supervisor
Supervisor is a process monitor for the Linux operating system, and
will automatically restart your queue:work process if it fails. To
install Supervisor on Ubuntu, you may use the following command:
sudo apt-get install supervisor
Configuring Supervisor
Supervisor configuration files are typically stored in the
/etc/supervisor/conf.d directory. Within this directory, you may
create any number of configuration files that instruct supervisor how
your processes should be monitored. For example, let's create a
laravel-worker.conf file that starts and monitors a queue:work
process:
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /home/forge/app.com/artisan queue:work sqs --sleep=3 --tries=3
autostart=true
autorestart=true
user=forge
numprocs=8
redirect_stderr=true
stdout_logfile=/home/forge/app.com/worker.log
stopwaitsecs=3600
In this example, the numprocs directive will instruct Supervisor to run 8 queue:work processes and monitor all of them, automatically
restarting them if they fail. You should change the queue:work sqs
portion of the command directive to reflect your desired queue
connection.
I'm using Beanstalkd as a work queue in my project.
Now, my project is completed and I have to deploy it on VPS(production server).
There is something that confusing me! should I ssh to production server and manually type php artisan queue:listen ? (it's crap)
Is there any server to run queue:listen as service?
You should use something like Supervisor to run the queue in production. This will allow you to run the process in the background, specify the number of workers you want processing queued jobs and restart the queue should the process fail.
As for the queue you choose to use, that's up to you. In the past I've used Beanstalkd locally installed on an instance and Amazon SQS. The local instance was fine for basic email sending and other async tasks, SQS was great for when the message volume was massive and needed to scale. There are other SaaS products too such as IronMQ, but the usual reason people run into issues in production are because they're not using Supervisor.
You can install Supervisor with apt-get. The following configuration is a good place to start:
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /home/forge/app.com/artisan queue:work --sleep=3 --tries=3
autostart=true
autorestart=true
numprocs=8
stdout_logfile=/home/user/app.com/worker.log
This will do the following:
Give the queue worker a unique name
Run the php artisan queue:work command
Automatically start the queue worker on system restart and automatically restart the queue workers should they fail
Run the queue worker across eight processes (this can be increased or reduced depending on your needs)
Log any output to /home/user/app.com/worker.log
To start Supervisor, you'd run the following (after re-reading the configuration/restarting):
sudo supervisorctl start laravel-worker:*
The documentation gives you some more in-depth information about using Supervisor to run Laravel's queue processes.
In my table "jobs" have 5 jobs, i used the process "php artisan queue:listen" on localhost then it ran all jobs and finished. But when i used this process on server then it only once ran one jobs (same "php artisan queue:work"). I use queue_driver is "database".
Update - Laravel 5.5+
Just use:
php artisan queue:work --once
TL;DR
php artisan queue:work = run one queued job and stop
php artisan queue:listen = run all queued jobs and listen for more until stopped
Further detail:
How are you running the commands? Are you just opening a terminal and running it or do you have a cron job to run at intervals?
If you're just opening a terminal and running php artisan queue:work will complete one task on the queue then stop, whereas if you run php artisan queue:listen it will process all the jobs in the queue and continue to listen for any further jobs until it is stopped.
Read further in the docs regarding queue:work:
Starting The Queue Listener
Laravel includes an Artisan command that will run new jobs as they are
pushed onto the queue. You may run the listener using the queue:listen
command:
php artisan queue:listen
Processing The First Job On The Queue
To process only the first job on the queue, you may use the queue:work command:
php artisan queue:work
What you should be doing?
I am guessing what you ideally want to do on your server is set up a cron job to run continuously at intervals and have it run queue:work. Better yet, acquaint yourself with the docs and make a decision after that.
See similar question answered here: What is the difference between queue:work --daemon and queue:listen
developers, I have a problem. My queue not working or i just not understand how it's works. I create a command which should add a new queue job. Driver for Queue is - database. After executing my command i see a new row in table 'jobs'. After that I try to do "php artisan queue:work" - but nothing happens.
Help me please, how can I execute this job?
From the documentation : [Daemon Queue Listener] The queue:work artisan command includes a --daemon option for forcing the queue worker to continue processing jobs without ever re-booting the framework. This results in a significant reduction of CPU usage when compared to the queue:listen command:
To start a queue worker in daemon mode, use the --daemon flag:
php artisan queue:work connection --daemon
However if you don't have multiple connections remove connection and execute it without connection :
php artisan queue:work --daemon
It worked for me.
Try
php artisan queue:listen
instead.
Yes there are times when your queue jobs won't run. For deployment if you are using redis queue driver, if not you can follow this here to install and configure redis and after which you should create a table for failed jobs using
php artisan queue:failed-table
php artisan migrate and then use php artisan queue:work redis --tries=3 --backoff=3 to retry every failed jobs 3 times after 3 seconds of failure.
To delay the next retry just add --delay=[NUM_OF_SECONDS] to your command.
For example, to wait 30 seconds to retry after failing just
run: php artisan queue:work tries=3 --delay=30
OR
php artisan queue:work --daemon --tries=3 --sleep=5 --delay=10
I have installed beanstaled and its working fine with laravel. The point where I am puzzled is that we have to do
php artisan queue:listen
to start listening queue. Right now, I am using it on amazone ec2 instance remotely through putty. but what is i close terminal? Will the jobs created through the code will work? Is it manually calling php artisan queue:listen or php artisan queue:work all time. Which does not seems fair.
If once php artisan queue:listen done, will it keep on running even if we close terminal?
Actually I dont know.
you need to install supervisor also. Here is a tutorial on using beanstalkd with laravel:
http://fideloper.com/ubuntu-beanstalkd-and-laravel4
Here are details on supervisor also:
http://supervisord.org/installing.html
I personally use a redis instance and run my queue with supervisor from there.
I find its a bit more memory effective then beanstalkd personally but each to there own.
Supervisor will execute the queue:listen command from artisan and this will run a job, if you have multiple supervisor processes then you can run multiple in line items.
depending on what you are doing i would almost look into python and multithereading also as i have used this for a few things i used to use a queue for and it has provided even better results.
example config file for supervisor:
[program:myqueue]
command=php artisan queue:listen --env=your_environment
directory=/path/to/laravel
stdout_logfile=/path/to/laravel/app/storage/logs/myqueue_supervisord.log
redirect_stderr=true
autostart=true
autorestart=true
You can also make use of Laravel's Task Scheduler i.e add the php artisan queue:listen command to the scheduler and sets its frequency to whatever you wants.
So that will make sure to call queue listen process automatically.
Hope it will make sense.