I have a Symfony2 app that under some circumstances has to send more than 10.000 push and email notifications.
I developed a SQS flow with some workers polling the queues to send emails and mobile push notifications.
But now, I have the problem that, when in the request/response cycle I need to send to SQS this task/jobs (maybe not that amount) this task itself is consuming a lot of time (response timeout is normally reached).
Should I process this task at background (I need to send back a quick response)? And how to handle possible errors with this scenario?
NOTE: Amazon SQS can receive 10 messages at one request and I already using this method. Maybe should I build a simple SQS Message with a lot of notifications jobs (max. 256K) to send less HTTP requests to SQS?
The moment you have a single action that triggers 10k actions, you need to try to find a way to tell the user that "OK, I got it. I'll start working on it and will let you know when it's done".
So to bring that work into the background, a domain event should be raised from your user's action which would be queued into SQS. The user gets notified, and then a worker can pick up that message from the queue and start sending emails and push notifications to another queue.
At the end of the day, 10k messages in batches of 10 are just 1k requests to SQS, which should be pretty quick anyway.
Try to keep your messages small. Don't send the whole content of an email into a queue message, because then you'll get unnecessary long latencies. Keep the content in a reachable place or just query for it again when consuming the message instead of passing big content up and down the network.
And how to handle possible errors with this scenario?
Amazon provides dead letter queues for this. In asynchronous systems I've built, I usually create a queue and then attach a redrive policy to it that says "if I see the same message on this queue 10 times, send it to a dead letter queue so that it doesn't bounce back and forth between the queue and a consumer for all eternity". The dead letter queue is simply another queue.
From a dead letter queue you can decide what to do with data that did not process. Since it's notifications (emails or push notifications) in your case, you might have another component in your system that will periodically reprocess a dead letter queue. Scheduled Lambdas are good for this.
Related
I was wondering, because I can not find anything on symfony or other resources, if php's symfony/messenger can handle messages in "bulk" with any async transport.
For example. Grab 20 messages from the bus, handle those 20 messages, and ack or reject any of the messages.
I know RabbitMQ has a feature to grab n-amount of messages from the queue, and process all of them in a single run.
In some cases this will have a better performance over scaling the async workers.
Does anybody have any leads, resources or experience with it? Or am I trying to resolve something by going against the idea of symfony/messenger?
[update]
I'm aware that bulk messages are not part of the (async) messaging concept. That each message should be processed individually. But some message brokers have implemented a feature to "grab" X-amount of messages from a queue and process them (either by sending an acknowledge or rejection, or otherwise). I know handling multiple messages in a single iteration increases complexity of any consumers, but in some cases it will improve performance.
I've used this concept of consuming multiple messages in a single iteration many times, but never with php's symfony/messenger.
This was not natively possible prior to symfony 5.4.
They added a BatchHandlerInterface which will allow you to batch (and choose the size of the batch) your messages.
You can find more info here :
Symfony - Handle messages in batches
GitHub PR of the feature
First I think you have the wrong concept. There is no such thing as “messages in bulk” in the queue world.
The idea of the queue is that one message is received in the consumer and the consumer is responsible of letting know the queue that the message was acknowledged, so it can be deleted. If this does not happen in X time the message is again visible for other messages.
If the messenger get 20 messages from the queue it still process them one by one and after he finish processing just acknowledge every message. These 20 messages are “hidden” to other consumers for some time /it depends of the configuration of the queue/. This answer also the question of multiple consumers.
I want to decouple PDF generation in order to avoid our customer waiting for it. I have been testing SNS but I read that message order is not guaranteed and I would need to combine it with SQS. I will have only one publisher and one receiver so it does not make sense to me having SNS in the midle. Instead of that I'm trying to create a Worker. My idea is to send the message directly to SQS attached to the Worker. I could create a cron.yml and check every 10 seconds or so, but it feels like a dirty solution.
I have been reading and trying for two days. I could not manage to make this work. I would like to send a message to the queue with the URL and JSON with data just to execute to http://localhost:80/pdf instead of http://localhost:80/ and generate PDF with the information stored in JSON.
Is what I want to achieve possible? I could not find a thing neither in the docs nor in forums. How the worker handle cron.yaml behind the scenes?
I am using the PHPMailer library to handle the sending of emails from within my application.
The problem is, when some emails are triggered to be sent (such as when a contact form has been submitted, a new user registers, etc), it could take 1-3 seconds for the page to load while the email is sending. If there is ever a problem sending the mail, the delay can be more.
I was thinking of saving any emails that need to be sent into a pending_emails table in my database, then just have a cron job ran every minute which would send out all those emails, then remove them from the table.
My question is, does this seem like a logical thing to do? Are there any potential resource concerns I should have with a cron job running every minute vs sending the email in runtime? (I need to run the cron job often, as someone may be waiting on an urgent message, for example "reset password" email)
You got everything right already.
Sending at runtime, just when you respond to the user's HTTP request, is the easiest thing to do. But the response is slowed down a bit by this, of course. That's not too bad in a small application, because sending email is faster than one might think. It definitely works.
Implementing a message queue is the more elegant and scalable approach, of course. But it takes a little more work. Your idea of using a pending_emails database table is totally valid. There are libraries and components for such queues, but you don't have to use them.
This is a very opinion based question so you're going to get a lot of different, conflicting answers because there are some who might tell you its ok to make a user wait 1-3 seconds since its not that long but I tend to disagree with that. What I typically do instead, however, is use a Queue.
There are ways to create a queue WITHOUT using 3rd party software, but there are some excellent tools out there such as RabbitMQ, Iron.io or Beanstalkd which can be extremely helpful to performing tasks in the background. These services push your task into a queue and these items in the queue are processed in a timely manner in the background, but the user gets an almost immediate response (depending on what you're doing). This is how I usually handle more resource intensive tasks, like sending an email, in the background to avoid holding up the response to a user.
Best of luck.
Look into threading (PHP Threading). I would suggest you create a new thread which invokes the sending of the email. This way, you can return a response to the user without waiting for the email to be sent, and the email sending process would run in parallel in another thread.
Has someone come up with an elegant way in PHP to recognize that SQS has just sent you the same message from the queue that it has sent previously? This could happen if the SQS has never received a 'deletemessage' request sent by the requestor because the original SQS queue could have gone down, so its replacement will resend the same message as per Amazon docs.
Would appreciate any pointers, code samples in PHP etc.
When you process a message you receive from SQS, there's a message id that is the same every time you receive the message, and a receipt handle (used to send the delete message) that changes each time. Saving the message ID along with the work you do as a result of the message allows your appplication to be aware that the work should not be repeated.
Of course, you need to set the visibility timeout on yoir queue high enough that you won't fail to delete each message or extend its invisibility before the timeout expires.
A recent addition to SQS is the ability to divert messages to a different queue -- they call it a "dead letter queue" -- once they've been delivered via the original queue a configurable number of times.
This seems like an easy enough workaround.
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/SQSDeadLetterQueue.html
It's always possible (though I have not caught it happening) that SQS could still deliver the message more than once, because, being a massive distributed syatem, SQS is designed to deliver messages "at least" once, so you have to accommodate that possibility. Saving the message ID in a column in my database that has a unique constraint on it, which makes inserting a duplicate impossible.
Amazon SQS is engineered to provide “at least once” delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.
https://aws.amazon.com/sqs/faqs/
Various users will submit their phone number to get notified for certain service that I provide. As soon as I start providing the service, a SMS notification should be sent out to all those users who had signed up(submitted their phone number).
foreach($phone_numbers as $number)
// api call to send the sms
$msg->send($number);
endforeach
Now the thing that is confusing me is, at some point I can have like 1000 users signed up, and when the scripts runs automatically when I add that service, it will consume a lot of time to loop through 1000 number and make 1000 API calls.
Is there a better way of doing it ?
One approach would be to use a message queue like RabbitMQ. That way, you could just push the task to the queue and have it handled by a separate process, which could run on a different server. It also has the advantage that it should be a bit more robust, since if it fails, your worker script can try again.
I haven't had occasion to use it with a PHP application, but I have used it with Celery for a Django app with good results. I found decent-looking PHP examples at http://blog.teqneers.com/2013/10/simple-spawn-rabbitmq-consumers-with-php/ and http://www.rabbitmq.com/tutorials/tutorial-two-php.html.