I have worked with Queues in laravel which will Queue the Jobs. php artisan queue:work command will take jobs in queue and process it.
But, i want to use pure queue without a job, One process will push itnto queue and another process read from queue and process it.
Eg, in a login api request, i will push just email id to queue. There will be a background process running(not queue:work) which can read the queue enties(list of emails) and do some action with it.
So the i need a custom worker which can read from queue.
How to do this in laravel?
Its not possible directly. https://github.com/laravel/framework/issues/17083
By design laravel doesnt allow this. You can however try to implement a custom solution.
Related
I was looking for a good way to manage a lot of background tasks, and i found out AWS SQS.
My software is coded in PHP. To complete a background task, the worker must be a CLI PHP application.
How am i thinking of acclompishing this with AWS SQS:
Client creates a message (message = task)
Message Added to Mysql DB
A Cron Job checks mysql db for messages and adds them to SQS queue
SQS Queue Daemon listents to queue for messages and sends HTTP POST requests to worker when a message is received
Worker receives POST request and forks a php shell_execute with parameters to do the work
Its neccessary to insert messages in MySQL because they are scheduled to be completed at a certain time
A little over complicated.
I need to know what is the best way to do this.
I would use AWS Lambda, with an SQS trigger to asynchronoulsy process messages dropped in the queue.
First, your application can post messages directly to SQS, there is no need to first insert the message in MySQL and have a separate daemon to feed the queue.
Secondly, you can write an AWS Lambda function in PHP, check https://aws.amazon.com/blogs/apn/aws-lambda-custom-runtime-for-php-a-practical-example/
Thirdly, I would wire the Lambda function to the queue, following this documentation : https://aws.amazon.com/blogs/apn/aws-lambda-custom-runtime-for-php-a-practical-example/
This will simplify your architecture (less moving parts, less code) and make it more scalable.
I understand the basic of Amazon SQS. Yet i'm still confused on how it runs? Is it an infinite running function that polls messages and deal with it? how would I achieve that in php?
What I have in mind is a cron job that triggers the polling and process the messages. is my understanding right?
There's more than one answer to this.
Yes, you could have cron poll regularly for new queue items. You could have a daemon running indefinitely (likely monitored by something like supervisor) that continues to poll in a loop.
There's also SQS triggers, where a new SQS item can automatically initiate something. There are multiple options available: new queue items can make an SNS notification, which could trigger a HTTP POST to a URL. They can also trigger a Lambda function.
I currently have a multi-server Laravel setup. I have multiple servers that are load balanced and share a database.
Each instance is also running a queue listener. I want to be able to dispatch two types of jobs:
A job that is only run once (e.g send email, update a model, etc)
A job that is run on ALL queue listeners (e.g delete a file from the filesystem)
The first I think is quite simple to implement, but unsure how to go about implementing the second one. Any ideas?
You can make a queue listener or worker handle specific queues. For example run a queue listener or worker to handle emails queue.
php artisan queue:listen --queue=emails
php artisan queue:work --queue=emails
You can now dispatch jobs to a this queue
dispatch((new Job)->onQueue('emails'));
This can help you setup multiple listeners/workers to handle different queues based on your requirements.
I'm working with external feeding API "Amazon mws" which I get all products for specific seller. Now let me say if I want to refresh these products by two methods: Automatically and manualy, the automatic approach would be refreshing this store every 12 hours for example, and the manual approach is to let the seller manually click a refresh link and further to display progress bar until this job is done.
So, now how can I manage to make these two methods ? I'm totally confused between jobs, queues and task scheduling, whether to use beanstalkd or redis ?
I just want somebody to direct me how to manage all of that and best practice for this situation... Thanks Artisans :)
I believe it is not possible to obtain inventory information just for one SKU from MWS Api. When we had similar requirement, we just created a php script that connects to MWS Reports api specifically used _GET_MERCHANT_LISTINGS_ALL_DATA_ report to download the report and insert/update into mysql database. We did not use redis or message queuing because, MWS Reports api works in such a way that you request for report and poll the report processing status. when it is success, download the report and process into database. we have been running this php script with cron every 30 mins.
For the automatic refresh you can run a task scheduler (system to system) the user is not involved, it is a perfect case for scheduling a task.
Whereas the refresh button, would be a job, but take this into account that a job can be queued or not queued, by either implementing shouldQueue or not. If you will like this job to be done in the background, you can queue that job to be done async.
Then set up an event that fires when the job has completed, or when the database is updated , and you can broadcast a notification to the user informing him her or team that the update has been completed.
So lets take it step by step, you can make jobs with artisan command this job you can dispatch from your controller.
Write your business logic in the job and implement shouldQueue. Job does not need a return statement. Then create the queue with artisan command this will create the queue table, and change queue driver in env to database (you can get quite a long way with database queue so you dont have to use beanstalkd, and it is good way of practice queues, you should then queue:listen ! Just a note when U use queue listen the listen will keep running untill you close the terminal, then when opening a terminal before running listen queue:restart.
Create the event you want with artisan and on your model listen for the event updated, when the updated is complete the event will fire.
Create the notification with artisan command and on the event listener, event notify. the notification you can customize what you want to notify.
You will need to broadcast this notification and for that you will need to create an account with pusher, and broadcast the event.
The laravel documentation covers it all but it is difficult to know where to start.
To broadcast with pusher install pusher and laravel echo then in jour event you broadcast to and a channel on your web routes will be created channel, there are some other settings and configs, just a tip to thest the broadcast and to receive something back on your front end just to test. broadcast to channel not private channel just a bit easier setting things up from there if it works do whatever you want.
Hope it helps.
#gustav1105 from laracasts
I am using Symfony framework3 with Pheanstalk bundle and Doctrine. I creating the event which sends data to beanstalk. The other SF app on the different server perform a job and update notification status on the first SF app to completed. How can I check when the status is updated and than set alert like that:
http://byrobin.nl/store/wp-content/uploads/sites/4/2016/03/local.png
I can create a command that have infinitive loop and checking for status update, maybe listener on preUpdate? Also I have the same problem with running command that checking and executing beanstalk jobs. In dev mode i run it by hand, but i try infinitive loop like while(true) but it load my buffer and crash. I was thinking of cron job that runs every minute or two? What is best solution for this two problems? Any advice?
1) It would be good with WebSockets as that doesn't involve while(true) loop. A websocket can be opened by the frontend after a task has been submitted for processing. After the job finished processing it would notify the server side of the websocket to relay the info back on the socket for the frontend.
2) Another option is to submit a message, and in the params name anonymous tube (make a unique name based on time and some prefix) where the worker needs to put the answer. And before submitting the job you subscribe on beanstalkd to the anonymous channel, then submit the job, and the job finishes it will post the answer to the tube. Since there is already a subscriber there it will reserve the job and deal with it, then delete it, and the tube gets removed too.