I'm working with external feeding API "Amazon mws" which I get all products for specific seller. Now let me say if I want to refresh these products by two methods: Automatically and manualy, the automatic approach would be refreshing this store every 12 hours for example, and the manual approach is to let the seller manually click a refresh link and further to display progress bar until this job is done.
So, now how can I manage to make these two methods ? I'm totally confused between jobs, queues and task scheduling, whether to use beanstalkd or redis ?
I just want somebody to direct me how to manage all of that and best practice for this situation... Thanks Artisans :)
I believe it is not possible to obtain inventory information just for one SKU from MWS Api. When we had similar requirement, we just created a php script that connects to MWS Reports api specifically used _GET_MERCHANT_LISTINGS_ALL_DATA_ report to download the report and insert/update into mysql database. We did not use redis or message queuing because, MWS Reports api works in such a way that you request for report and poll the report processing status. when it is success, download the report and process into database. we have been running this php script with cron every 30 mins.
For the automatic refresh you can run a task scheduler (system to system) the user is not involved, it is a perfect case for scheduling a task.
Whereas the refresh button, would be a job, but take this into account that a job can be queued or not queued, by either implementing shouldQueue or not. If you will like this job to be done in the background, you can queue that job to be done async.
Then set up an event that fires when the job has completed, or when the database is updated , and you can broadcast a notification to the user informing him her or team that the update has been completed.
So lets take it step by step, you can make jobs with artisan command this job you can dispatch from your controller.
Write your business logic in the job and implement shouldQueue. Job does not need a return statement. Then create the queue with artisan command this will create the queue table, and change queue driver in env to database (you can get quite a long way with database queue so you dont have to use beanstalkd, and it is good way of practice queues, you should then queue:listen ! Just a note when U use queue listen the listen will keep running untill you close the terminal, then when opening a terminal before running listen queue:restart.
Create the event you want with artisan and on your model listen for the event updated, when the updated is complete the event will fire.
Create the notification with artisan command and on the event listener, event notify. the notification you can customize what you want to notify.
You will need to broadcast this notification and for that you will need to create an account with pusher, and broadcast the event.
The laravel documentation covers it all but it is difficult to know where to start.
To broadcast with pusher install pusher and laravel echo then in jour event you broadcast to and a channel on your web routes will be created channel, there are some other settings and configs, just a tip to thest the broadcast and to receive something back on your front end just to test. broadcast to channel not private channel just a bit easier setting things up from there if it works do whatever you want.
Hope it helps.
#gustav1105 from laracasts
Related
I am building a Multi-Tenant web application using Laravel/PHP that will be hosted on AWS as SaaS at the end. I have around 15-20 different background jobs that need scheduling for each tenant. The jobs need to be fired every 5 minutes as well. Thus the number of jobs which need to be fired for 100 tenants would be around 2000. I am left with 2 challenges in achieving this
Is there a cloud solution that distributes and manages the load of the scheduled jobs automatically?
If one is out there, how can we create those 15+ scheduled jobs on the fly? Is there an API available?
Looking for your assistance
Finally, I have found a solution to my problem.
We cannot scale the background jobs in the way I want. It required me to look into the solution from a completely different angle.
The ideal solution to my problem is that I should generate SQS messages (with a payload describing the tenant id, the job needs to be executed and any additional parameters) corresponding to the number of tenants on a set interval and queue it.
For example, if I have 100 tenants and I want to run "Job 1" every our, the main application will generate 100 SQS messages and queue it in a particular SQS Queue every hour. It will do the same for all 15 different jobs I have per tenant.
On the other end, a scalable AWS Lambda function listening to the SQS queue will pick up the payload and execute the intended task based on the data being carried by the payload.
But unfortunately, my expertise lies in PHP/Laravel technology which is still not in the AWS Lambda stack. Hence I figured out a workaround as follows.
I built a Docker image with my PHP/Laravel application and placed it in Amazon ECS (EC2 container service). Still, I have the AWS Lambda function in place but this time it acts as a trigger to my docker containers. The Lambda picks an SQS Message, processes the payload and spawns a Docker container on ECS based on my Docker image. I got some of the ideas from the following article to arrive at this solution.
https://aws.amazon.com/blogs/compute/better-together-amazon-ecs-and-aws-lambda/
Laravel has option to schedule Task/Jobs:
Refer: https://laravel.com/docs/6.x/scheduling
so you can keep jobs of your client in your database and than do it some like below:
Scheduling Queued Jobs
The job method may be used to schedule a queued job. This method provides a convenient way to schedule jobs without using the call method to manually create Closures to queue the job:
$schedule->job(new ClientJob)->everyFiveMinutes();
// Dispatch the job to the "clientjob" queue...
$schedule->job(new ClientJob, 'clientjob')->everyFiveMinutes();
or
Scheduling Shell Commands
The exec method may be used to issue a command to the operating system:
$schedule->exec('node /home/forge/script.js')->everyFiveMinutes();
In my Laravel 5.4 web app user can request report generation that takes a couple of minutes due to a big amount of data. Because of these he couldn't work with application no more, until report will be generated. To fix this problem I have read about queues in laravel and separated out my report generation code to the job class, but my app still holds until report will be generated. How can I fix that?
To be absolutely clear I will sum up my problem:
User make request for report generation (my app absolutely holds at this moment)
My app receives POST request in routes and calls a function from the controller class.
Controller's function dispatches a job, that should generate report and put it into the client web folder.
It sounds like you have already pretty much solved the problem by introducing a queue. Put the job in the queue, but don't keep track of its progress - allow your code to continue and return to the user. It should be possible to "fire-and-forget", and then either ask the user to check if the report is ready in a couple of minutes, or offer the ability to email it to them when it is completed.
By default, Laravel uses the sync queue driver. This driver executes the queued jobs in the same request as the one they are created in. So this won't make any difference.
You should take a look at other drivers and use the Laravel queue worker background process to execute jobs to make sure they don't hold the webrequest from completing.
I have worked with Queues in laravel which will Queue the Jobs. php artisan queue:work command will take jobs in queue and process it.
But, i want to use pure queue without a job, One process will push itnto queue and another process read from queue and process it.
Eg, in a login api request, i will push just email id to queue. There will be a background process running(not queue:work) which can read the queue enties(list of emails) and do some action with it.
So the i need a custom worker which can read from queue.
How to do this in laravel?
Its not possible directly. https://github.com/laravel/framework/issues/17083
By design laravel doesnt allow this. You can however try to implement a custom solution.
I am trying to set up an API system that synchronously communicates with a number of workers in Laravel. I use Laravel 5.4 and, if possible, would like to use its functionality whenever possible without too many plugins.
What I had in mind are two servers. The first one with a Laravel instance – let’s call it APP – receiving and answering requests from and to a user. The second one runs different workers, each a Laravel instance. This is how I see the workflow:
APP receives a request from user
APP puts request on a queue
Workers look for jobs on the queue and eventually finds one.
Worker resolves job
Worker responses to APP OR APP finds out somehow that job is resolved
APP sends response to user
My first idea was to work with queues and beanstalkd. The problem is that this all seem to work asynchronously. Is there a way for the APP to wait for the result of one of the workers?
After some more research I stumbled upon Guzzle. Would this be a way to go?
EDIT: Some extra info on the project.
I am talking about a Restful API. E.g. a user sends a request in the form of "https://our.domain/article/1" and their API token in the header. What the user receives is a JSON formatted string like {"id":1,"name":"article_name",etc.}
The reason for using two sides is twofold. At one hand there is the use of different workers. On the other hand we want all the logic of the API as secure as possible. When a hack attack is made, only the APP side would be compromised.
Perhaps I am making things all to difficult with the queues and all that? If you have a better approach to meet the same ends, that would of course also help.
I know your question was how you could run this synchronously, I think that the problem that you are facing is that you are not able to update the first server after the worker is done. The way you could achieve this is with broadcasting.
I have done something similar with uploads in our application. We use a Redis queue but beanstalk will do the same job. On top of that we use pusher which the uses sockets that the user can subscribe to and it looks great.
User loads the web app, connecting to the pusher server
User uploads file (at this point you could show something to tell the user that the file is processing)
Worker sees that there is a file
Worker processes file
Worker triggers and event when done or on fail
This event is broadcasted to the pusher server
Since the user is listening to the pusher server the event is received via javascript
You can now show a popup or update the table with javascript (works even if the user has navigated away)
We used pusher for this but you could use redis, beanstalk and many other solutions to do this. Read about Event Broadcasting in the Laravel documentation.
I am using Symfony framework3 with Pheanstalk bundle and Doctrine. I creating the event which sends data to beanstalk. The other SF app on the different server perform a job and update notification status on the first SF app to completed. How can I check when the status is updated and than set alert like that:
http://byrobin.nl/store/wp-content/uploads/sites/4/2016/03/local.png
I can create a command that have infinitive loop and checking for status update, maybe listener on preUpdate? Also I have the same problem with running command that checking and executing beanstalk jobs. In dev mode i run it by hand, but i try infinitive loop like while(true) but it load my buffer and crash. I was thinking of cron job that runs every minute or two? What is best solution for this two problems? Any advice?
1) It would be good with WebSockets as that doesn't involve while(true) loop. A websocket can be opened by the frontend after a task has been submitted for processing. After the job finished processing it would notify the server side of the websocket to relay the info back on the socket for the frontend.
2) Another option is to submit a message, and in the params name anonymous tube (make a unique name based on time and some prefix) where the worker needs to put the answer. And before submitting the job you subscribe on beanstalkd to the anonymous channel, then submit the job, and the job finishes it will post the answer to the tube. Since there is already a subscriber there it will reserve the job and deal with it, then delete it, and the tube gets removed too.