I am building a Multi-Tenant web application using Laravel/PHP that will be hosted on AWS as SaaS at the end. I have around 15-20 different background jobs that need scheduling for each tenant. The jobs need to be fired every 5 minutes as well. Thus the number of jobs which need to be fired for 100 tenants would be around 2000. I am left with 2 challenges in achieving this
Is there a cloud solution that distributes and manages the load of the scheduled jobs automatically?
If one is out there, how can we create those 15+ scheduled jobs on the fly? Is there an API available?
Looking for your assistance
Finally, I have found a solution to my problem.
We cannot scale the background jobs in the way I want. It required me to look into the solution from a completely different angle.
The ideal solution to my problem is that I should generate SQS messages (with a payload describing the tenant id, the job needs to be executed and any additional parameters) corresponding to the number of tenants on a set interval and queue it.
For example, if I have 100 tenants and I want to run "Job 1" every our, the main application will generate 100 SQS messages and queue it in a particular SQS Queue every hour. It will do the same for all 15 different jobs I have per tenant.
On the other end, a scalable AWS Lambda function listening to the SQS queue will pick up the payload and execute the intended task based on the data being carried by the payload.
But unfortunately, my expertise lies in PHP/Laravel technology which is still not in the AWS Lambda stack. Hence I figured out a workaround as follows.
I built a Docker image with my PHP/Laravel application and placed it in Amazon ECS (EC2 container service). Still, I have the AWS Lambda function in place but this time it acts as a trigger to my docker containers. The Lambda picks an SQS Message, processes the payload and spawns a Docker container on ECS based on my Docker image. I got some of the ideas from the following article to arrive at this solution.
https://aws.amazon.com/blogs/compute/better-together-amazon-ecs-and-aws-lambda/
Laravel has option to schedule Task/Jobs:
Refer: https://laravel.com/docs/6.x/scheduling
so you can keep jobs of your client in your database and than do it some like below:
Scheduling Queued Jobs
The job method may be used to schedule a queued job. This method provides a convenient way to schedule jobs without using the call method to manually create Closures to queue the job:
$schedule->job(new ClientJob)->everyFiveMinutes();
// Dispatch the job to the "clientjob" queue...
$schedule->job(new ClientJob, 'clientjob')->everyFiveMinutes();
or
Scheduling Shell Commands
The exec method may be used to issue a command to the operating system:
$schedule->exec('node /home/forge/script.js')->everyFiveMinutes();
Related
I was looking for a good way to manage a lot of background tasks, and i found out AWS SQS.
My software is coded in PHP. To complete a background task, the worker must be a CLI PHP application.
How am i thinking of acclompishing this with AWS SQS:
Client creates a message (message = task)
Message Added to Mysql DB
A Cron Job checks mysql db for messages and adds them to SQS queue
SQS Queue Daemon listents to queue for messages and sends HTTP POST requests to worker when a message is received
Worker receives POST request and forks a php shell_execute with parameters to do the work
Its neccessary to insert messages in MySQL because they are scheduled to be completed at a certain time
A little over complicated.
I need to know what is the best way to do this.
I would use AWS Lambda, with an SQS trigger to asynchronoulsy process messages dropped in the queue.
First, your application can post messages directly to SQS, there is no need to first insert the message in MySQL and have a separate daemon to feed the queue.
Secondly, you can write an AWS Lambda function in PHP, check https://aws.amazon.com/blogs/apn/aws-lambda-custom-runtime-for-php-a-practical-example/
Thirdly, I would wire the Lambda function to the queue, following this documentation : https://aws.amazon.com/blogs/apn/aws-lambda-custom-runtime-for-php-a-practical-example/
This will simplify your architecture (less moving parts, less code) and make it more scalable.
Hoping I can get some help (again). I'm working on a multi-tenant PHP application. Each tenant will have their own database (mysql). Ultimately, my plan is to stand the service up on AWS using ELB, EC2, and DynamoDB.
However, the app will need to run certain scheduled tasks (it needs to pull open invoices from a PSA for certain customers, and then charge the customer using Authroize.net CIM and mark it paid in the PSA).
For a regular application, I would simply create a cron script that runs daily to create/process the payment batches. I'm just not sure what the appropriate approach would be to run the cron across each tenant (for each database). Maybe one master cron job that runs across each tenant, or do I write a script to create / maintain cron jobs for each tenant using SWF?
Thanks for your input.
I've had reasonable success with doing batch processing via cron in the past. It might be helpful to record tenant creation in a table that you can query as a source for which databases to run against within the cron job.
I'm working with external feeding API "Amazon mws" which I get all products for specific seller. Now let me say if I want to refresh these products by two methods: Automatically and manualy, the automatic approach would be refreshing this store every 12 hours for example, and the manual approach is to let the seller manually click a refresh link and further to display progress bar until this job is done.
So, now how can I manage to make these two methods ? I'm totally confused between jobs, queues and task scheduling, whether to use beanstalkd or redis ?
I just want somebody to direct me how to manage all of that and best practice for this situation... Thanks Artisans :)
I believe it is not possible to obtain inventory information just for one SKU from MWS Api. When we had similar requirement, we just created a php script that connects to MWS Reports api specifically used _GET_MERCHANT_LISTINGS_ALL_DATA_ report to download the report and insert/update into mysql database. We did not use redis or message queuing because, MWS Reports api works in such a way that you request for report and poll the report processing status. when it is success, download the report and process into database. we have been running this php script with cron every 30 mins.
For the automatic refresh you can run a task scheduler (system to system) the user is not involved, it is a perfect case for scheduling a task.
Whereas the refresh button, would be a job, but take this into account that a job can be queued or not queued, by either implementing shouldQueue or not. If you will like this job to be done in the background, you can queue that job to be done async.
Then set up an event that fires when the job has completed, or when the database is updated , and you can broadcast a notification to the user informing him her or team that the update has been completed.
So lets take it step by step, you can make jobs with artisan command this job you can dispatch from your controller.
Write your business logic in the job and implement shouldQueue. Job does not need a return statement. Then create the queue with artisan command this will create the queue table, and change queue driver in env to database (you can get quite a long way with database queue so you dont have to use beanstalkd, and it is good way of practice queues, you should then queue:listen ! Just a note when U use queue listen the listen will keep running untill you close the terminal, then when opening a terminal before running listen queue:restart.
Create the event you want with artisan and on your model listen for the event updated, when the updated is complete the event will fire.
Create the notification with artisan command and on the event listener, event notify. the notification you can customize what you want to notify.
You will need to broadcast this notification and for that you will need to create an account with pusher, and broadcast the event.
The laravel documentation covers it all but it is difficult to know where to start.
To broadcast with pusher install pusher and laravel echo then in jour event you broadcast to and a channel on your web routes will be created channel, there are some other settings and configs, just a tip to thest the broadcast and to receive something back on your front end just to test. broadcast to channel not private channel just a bit easier setting things up from there if it works do whatever you want.
Hope it helps.
#gustav1105 from laracasts
I see a common pattern for services that we try to develop and I wonder if there are tools / libraries out there that would help here. While the default jobs as discussed in microservice literature is from the REQUEST -> RESPONSE nature, our jobs are more or less assignments of semi permanent tasks.
Examples of such tasks
Listen on the message queue for data from source X and Y, correlate the data that comes in and store it in Z.
Keep an in-memory buffer that calculates a running average of the past 15 mins of data everytime a new data entry comes in.
Currently our services are written in PHP. Due to the perceived overhead of PHP processes and connections to the message queue we'd like a single service process to handle multiple of those jobs simultanously.
A chart that hopefully illustrated the setup that we have in our head:
Service Workers are currently deamonized PHP scripts
For the Service Registry we are looking at Zookeeper
While Zookeeper (and Curator) do loadbalancing, I did not find anything around distributing permanent jobs (that are updatable, removable, and must be reassigned when a worker dies)
Proposed responsibilities of a Job Manager
Knows about jobs
Knows about services that can do these jobs
Can assign jobs to services
Can send job updates to services
Can reassign jobs if a worker dies
Are there any libraries / tools that can tackle such problems, and can thus function as the Job Manager? Or is this all one big anti pattern and should we do it some other way?
You should have a look at Gearman.
It composes of a client which assigns the jobs, one or more workers which will pick up and execute the jobs and a server which will maintain the list of functions (services) and jobs pending. It will re-assign the jobs if a worker dies.
Your workers sound like (api-less) services itself. So, your requirements can be reformulated as:
Knows about deployed services
Knows about nodes that can host there services
Can deploy services to nodes
Can [send job updates to services] = redeploy services/invoke some API on deployed services
Can redeploy service if service or node dies
Look at Docker to deploy, run and manage isolated processes on host.
RabbitMq is simple message queue that is fairly easy to get going with.
I'm looking for a solution to add items into a queue and execute them one-by-one in a similar method to google appengine's tasks manager. Each task will be executed using a http request to a php script.
As i'm using amazon, i understood that the best practice is using the SNS service that will be responsible for receiving new tasks, adding them to a queue (Amazon's SQS service) and also inform my php worker that a new task has been pushed into the queue so he can look for it and execute it.
There are several issues with that method (like the need to limit the number of workers instances via the worker itself or just the possibility that the task won't be in the queue when we call the worker because we add the task to the queue in the same time).
I would like to hear if there are any better options or a nicer way of implementing a tasks manager. I preffer using the amazon's services but i'm open to any new suggestion, looking for the best method. Features that are missing in amazon like FIFO and priorities support would also be a nice addition.
Thanks!
Ben
I have found a good solution.
AWS Beanstalk service is apparently offering an option to define a new elastic-beanstalk instance as a "worker" or a "web server". in case you define it as a "Worker", you'll be able to attach it to a sqs queue and it will be responsible for polling the queue and performing the task (with the code you deploy to the instance).