Our web application running on elastic beanstalk logs activity of incoming request to a database. We want to decouple the dB logging from the request processing path, so that response time can be sped up. We decided to use sqs queues and beanstalk worker. The idea is to queue the logging event to sqs, and have the worker receive the events and let it do the logging to the dB.
Now the need is to optimize the dB logging operation and avoid creating one connection per message in the queue. From my understanding the sqs daemon would Call the worker for each message, is there a way to have the daemon send messages in a batch, so that there's only one message and it's body has contents of all messages?
Or do we need to use a secondary queue or write a custom sqs message aggregator that processes n messages from the queue and then sends one batch message to another queue and that then gets written to the dB once?
We are using php and mysql
From my experience, defaultly you cannot. The daemon calls your application for each message.
What you can do might be that you cache the messages locally (assuming you are using single instance instead of auto scaling one) in a file (locking system for multi-processing) and then uses the scheduling of ELB cronjob to retrieve information from the file then do your DB operations every a certain amount of time. Thus, you can do that DB operation in a batch.
If you want to use auto scaling with multiple instances, you might need to use another messaging which is a waste compared with another option. This option is you write your own code using based off aws sdk to receive/delete from SQS in a batch and then update your database.
Related
I am building a Multi-Tenant web application using Laravel/PHP that will be hosted on AWS as SaaS at the end. I have around 15-20 different background jobs that need scheduling for each tenant. The jobs need to be fired every 5 minutes as well. Thus the number of jobs which need to be fired for 100 tenants would be around 2000. I am left with 2 challenges in achieving this
Is there a cloud solution that distributes and manages the load of the scheduled jobs automatically?
If one is out there, how can we create those 15+ scheduled jobs on the fly? Is there an API available?
Looking for your assistance
Finally, I have found a solution to my problem.
We cannot scale the background jobs in the way I want. It required me to look into the solution from a completely different angle.
The ideal solution to my problem is that I should generate SQS messages (with a payload describing the tenant id, the job needs to be executed and any additional parameters) corresponding to the number of tenants on a set interval and queue it.
For example, if I have 100 tenants and I want to run "Job 1" every our, the main application will generate 100 SQS messages and queue it in a particular SQS Queue every hour. It will do the same for all 15 different jobs I have per tenant.
On the other end, a scalable AWS Lambda function listening to the SQS queue will pick up the payload and execute the intended task based on the data being carried by the payload.
But unfortunately, my expertise lies in PHP/Laravel technology which is still not in the AWS Lambda stack. Hence I figured out a workaround as follows.
I built a Docker image with my PHP/Laravel application and placed it in Amazon ECS (EC2 container service). Still, I have the AWS Lambda function in place but this time it acts as a trigger to my docker containers. The Lambda picks an SQS Message, processes the payload and spawns a Docker container on ECS based on my Docker image. I got some of the ideas from the following article to arrive at this solution.
https://aws.amazon.com/blogs/compute/better-together-amazon-ecs-and-aws-lambda/
Laravel has option to schedule Task/Jobs:
Refer: https://laravel.com/docs/6.x/scheduling
so you can keep jobs of your client in your database and than do it some like below:
Scheduling Queued Jobs
The job method may be used to schedule a queued job. This method provides a convenient way to schedule jobs without using the call method to manually create Closures to queue the job:
$schedule->job(new ClientJob)->everyFiveMinutes();
// Dispatch the job to the "clientjob" queue...
$schedule->job(new ClientJob, 'clientjob')->everyFiveMinutes();
or
Scheduling Shell Commands
The exec method may be used to issue a command to the operating system:
$schedule->exec('node /home/forge/script.js')->everyFiveMinutes();
I was looking for a good way to manage a lot of background tasks, and i found out AWS SQS.
My software is coded in PHP. To complete a background task, the worker must be a CLI PHP application.
How am i thinking of acclompishing this with AWS SQS:
Client creates a message (message = task)
Message Added to Mysql DB
A Cron Job checks mysql db for messages and adds them to SQS queue
SQS Queue Daemon listents to queue for messages and sends HTTP POST requests to worker when a message is received
Worker receives POST request and forks a php shell_execute with parameters to do the work
Its neccessary to insert messages in MySQL because they are scheduled to be completed at a certain time
A little over complicated.
I need to know what is the best way to do this.
I would use AWS Lambda, with an SQS trigger to asynchronoulsy process messages dropped in the queue.
First, your application can post messages directly to SQS, there is no need to first insert the message in MySQL and have a separate daemon to feed the queue.
Secondly, you can write an AWS Lambda function in PHP, check https://aws.amazon.com/blogs/apn/aws-lambda-custom-runtime-for-php-a-practical-example/
Thirdly, I would wire the Lambda function to the queue, following this documentation : https://aws.amazon.com/blogs/apn/aws-lambda-custom-runtime-for-php-a-practical-example/
This will simplify your architecture (less moving parts, less code) and make it more scalable.
I am using Symfony framework3 with Pheanstalk bundle and Doctrine. I creating the event which sends data to beanstalk. The other SF app on the different server perform a job and update notification status on the first SF app to completed. How can I check when the status is updated and than set alert like that:
http://byrobin.nl/store/wp-content/uploads/sites/4/2016/03/local.png
I can create a command that have infinitive loop and checking for status update, maybe listener on preUpdate? Also I have the same problem with running command that checking and executing beanstalk jobs. In dev mode i run it by hand, but i try infinitive loop like while(true) but it load my buffer and crash. I was thinking of cron job that runs every minute or two? What is best solution for this two problems? Any advice?
1) It would be good with WebSockets as that doesn't involve while(true) loop. A websocket can be opened by the frontend after a task has been submitted for processing. After the job finished processing it would notify the server side of the websocket to relay the info back on the socket for the frontend.
2) Another option is to submit a message, and in the params name anonymous tube (make a unique name based on time and some prefix) where the worker needs to put the answer. And before submitting the job you subscribe on beanstalkd to the anonymous channel, then submit the job, and the job finishes it will post the answer to the tube. Since there is already a subscriber there it will reserve the job and deal with it, then delete it, and the tube gets removed too.
I'm writing a web app in PHP + Laravel + MySQL.
In the system, a user can schedule emails (and other API calls) at arbitrary times (much like how you schedule posts in WordPress). I can use CRON to inspect the database every 5min or so to find emails that should be sent, send them, and update their status.
However, this is a SaaS app. So the amount of emails to be sent at a particular time can grow rapidly. I can create a "lock file" every time the CRON script runs so that only one instance of it is running at a time. The lock file will be deleted after a script finishes execution.
But with potentially large data, I would want a way to process multiple messages simultaneously, potentially using multiple "workers." Is there any existing solution manage such a queue?
Yes! Task/Message/Job queues are what you are looking for! They allow you to put various tasks in queues from which you can retrieve them and process them, this process can scale horizontally as each worker can pull a task once its finished with the previous one.
You should have the cron maybe every minute/two minutes that just uploads the task and what needs to be done. This will make sure the cron is very quick.
Take a look at Iron.io Here is an extract from the website which gives a nice overview of these kinds of systems:
An easy-to-use scalable task queue that gives cloud developers a
simple way to offload front-end tasks, run scheduled jobs, and process
tasks in the background and at scale.
Gearman is also a great solution that you can use yourself and is very simple. You can send the message in many different languages and use a different langauge to process it. Say PHP -> C etc...
The Wikipedia link will tell you everything you need to know, here is a quick excerpt:
Message queues provide an asynchronous communications protocol,
meaning that the sender and receiver of the message do not need to
interact with the message queue at the same time. Messages placed onto
the queue are stored until the recipient retrieves them.
I have a scenario where I have drop message to queue and fetch this message from other process and do the stuff.
I have a website written on PHP, I am reading and writing to Redis while main database is MySQL.
I don't want to delay user response time, so i am using Redis. After writing to Redis i want to drop a message in queue and then other process running will read it and store the transaction in database. So sending a message to queue while writing to Redis is not a problem as this can be easily done in PHP, reading from queue can also be achieved by running a PHP script in domain (with open socket), cron etc.
I need to know if there is any opensource software available which can read the message from queue as soon as they arrive and trigger a PHP script with parameters. This mechanism will be fast.
I am not sure about the efficiency of PHP socket running as domain, but for cron there is certain delay.