We have a specific problem with our RabbitMQ
We need to ensure that over many Queues with different topics there is a 1-1 and not 1-N messages.
Just like in MQTT when one consumer acknowledges the push message it goes away.
The problem is that topics are an fan-out type.
Example:
one script publishes a message with routing key:
local.appNAme.clientName
And we have 2 ques with folowing rounting keys
que 1. local.*.clientName
que 2. local.#
They both get the Quqe as far i get it. And acknowledging one unlike in MQTT does not delete the other one.
Any idea how to implement the same behavior with routing wildcards in Rabbit?
ps. They are on the same server on the same Exchange.
Related
I was wondering, because I can not find anything on symfony or other resources, if php's symfony/messenger can handle messages in "bulk" with any async transport.
For example. Grab 20 messages from the bus, handle those 20 messages, and ack or reject any of the messages.
I know RabbitMQ has a feature to grab n-amount of messages from the queue, and process all of them in a single run.
In some cases this will have a better performance over scaling the async workers.
Does anybody have any leads, resources or experience with it? Or am I trying to resolve something by going against the idea of symfony/messenger?
[update]
I'm aware that bulk messages are not part of the (async) messaging concept. That each message should be processed individually. But some message brokers have implemented a feature to "grab" X-amount of messages from a queue and process them (either by sending an acknowledge or rejection, or otherwise). I know handling multiple messages in a single iteration increases complexity of any consumers, but in some cases it will improve performance.
I've used this concept of consuming multiple messages in a single iteration many times, but never with php's symfony/messenger.
This was not natively possible prior to symfony 5.4.
They added a BatchHandlerInterface which will allow you to batch (and choose the size of the batch) your messages.
You can find more info here :
Symfony - Handle messages in batches
GitHub PR of the feature
First I think you have the wrong concept. There is no such thing as “messages in bulk” in the queue world.
The idea of the queue is that one message is received in the consumer and the consumer is responsible of letting know the queue that the message was acknowledged, so it can be deleted. If this does not happen in X time the message is again visible for other messages.
If the messenger get 20 messages from the queue it still process them one by one and after he finish processing just acknowledge every message. These 20 messages are “hidden” to other consumers for some time /it depends of the configuration of the queue/. This answer also the question of multiple consumers.
There is a long running process(Excel report creation) in my web app that needs to be executed in a background.
Some details about the app and environment.
The app consists of many instances, where each client has separate one (with customized business logic) while everything is hosted on our server. The functionality that produces Excel is the same.
I'm planning to have one rabbitMq server installed. One part of app(Publisher) will take all report options from user and will put it into message. And some background job(Consumer) will consume it, produce report and send it via email.
However, there is a flaw in such design, where,say, users from one instance will queue lots of complicated reports(worth ~10 min of work) and a user from another instance will queue an easy one(1-2 mins) and he will have to wait until others will finish.
There could be separate queues for each app instance, but in that case I would need to create one consumer per instance. Given that there are 100+ instances atm, it doesn't look like a viable approach.
I was thinking if it's possible to have a script that checks all available queues(and consumers) and create a new consumer for a queue that doesn't have one. There are no limitations on language for consumer and such script.
Does that sound like a feasible approach? If not, please give a suggestion.
Thanks
As I understood topic correctly everything lies on one server - RabbitMQ, web application, different instances per client and messeges' consumers. In that case I rather put different topics per message (https://www.rabbitmq.com/tutorials/tutorial-five-python.html) and introduce consumer priorities (https://www.rabbitmq.com/consumer-priority.html). Based on that options during publishing of the message I will create combination of topic and priority of the message - publisher will know number of already sent reports per client, selected options and will decide is it high, low or normal priority.
Logic to pull messages based on that data will be in the consumer so consumer will not get heavy topics when there are in process already 3 (example).
Based on the total number of messages in the queue (its not accurate 100%) and previous topics and priorities you can implement kind of leaking bucket strategy in order to get control of resources- max 100 number of reports generated simultaneously.
You can consider using ZeroMQ (http://zeromq.org) for your case its maybe more suitable that RabbitMQ because is more simple and its broker less solution.
I have a question concerning the third RabbitMQ tutorial. I am trying to implement something similar, except there is no guarantee the consumer(s) would be running at the time the producer sends a message to the exchange.
So, I have my producer which publishes the messages to a fanout exchange:
$channel->exchange_declare('my_exchange', 'fanout', false, false, false);
$channel->basic_publish('my_message', 'my_exchange');
In my publishers, I declare queues, which I then bind to the exchange:
list($queueName,, ) = $channel->queue_declare("", false, false, true, false);
$channel->queue_bind($queueName, 'my_exchange');
And this is where my problem has it's root. The tutorial says:
The messages will be lost if no queue is bound to the exchange yet,
but that's okay for us; if no consumer is listening yet we can safely
discard the message.
Is there a way to somehow preserve those messages, so when a consumer starts, it would access the previously sent messages? The only way I figured out how to do it is to declare the same queue in my producer and my publisher, but it kind of defeats the purpose of having an exchange and separate queues for different consumers.
The queues need to exist, doesn't matter really who/what creates them: it can be producer (althoug I would strongly discourage this), consumer, some third admin app that just creates queus via rest api, rabbitmqctl... If you want to consume the queue(s) later, just make sure that they're durable and that TTL for messages is long enough (also durable messages if needed). But beware that your queue(s) don't get into flow state.
The only way I figured out how to do it is to declare the same queue
in my producer and my publisher, but it kind of defeats the purpose of
having an exchange and separate queues for different consumers.
First - I think you meant to say in my producer and my subscriber :)
Second, separate queues for consumers (or queue per consumer) is just in this example. Bare in mind that this is for a fanout exchange, and each consumer decalres an exclusive queue - when the consumer disconnects, the queue is gone. And that's why that's okay for us, because we're simply broadcasting and who wants the broadcast (the messages) needs to get it. Fanout exchange just puts messages to all the queues bound to it, that's it. It's perfectly ok to have multiple consumers consuming from same queue (look at tutorial 2).
So you just need to consider your use case. Of course it doesn't make sense to create fanout exchange and pre-setup the queus for the consumers... Perhaps you need just some routing keys or something else.
In this example (so tutorial 3) it's ment that there is a brodcast of messages, and if no one get's them, not a big (or small) deal. If anyone wants them, they need to get them. It's like a tv channel - regardless if someone is watching or not, the signal goes on.
Consumers should attach themselves to queues, they shouldn't declare their own queues. Think of queues as buckets of work to be done. Depending on the workload you can add N consumers to those queues to do the work.
When you create an exchange you should have one or more queues (buckets of work) that are attached to that exchange. If you do this, messages will flow into the queues and start to queue-up (pardon the pun). Your consumers can then attach whenever they are ready and start doing the work.
I'm a little confused. I'm trying to implement topic exchanges and am not sure what is needed.
I want to have several routing keys and 1 topic exchange (the default amq.topic). My keys would be like:
customer.appA.created
customer.appB.created
customer.*.created
I want my queue(s) to be durable, but do I need 1 'customer' queue or 2 queues for appA and appB?
I have my publisher figured out; connect, exchange declare, basic publish.
But I'm struggling with the consumers. Let's say I want to open 3 consoles, one for each of the aforementioned routing keys.
My current consumer has: connect, exchange declare, queue bind, basic consume. These are connected to a durable 'customer' queue. However my messages are being round-robin'ed to each console/consumer and not using the routing keys.
So my questions;
For a typical topic exchange set up; how many queues do you need?
Can my consumers get away with just exchange binding, or does it have to include queue interaction?
Is it possible for a single message to appear in 2 consumers with topic exchange (or do you need fanout for that)?
First thing first : Exchange do not deliver to Consumer. It delivers messages to Queue for matching routing kyes.
1. For a typical topic exchange set up; how many queues do you need?
If you have multiple consumer then you will need one queue for every consumer.
2. Can my consumers get away with just exchange binding, or does it have to include queue interaction?
You need to bind consumer with queue, if queue not exist then create and bind.
3. Is it possible for a single message to appear in 2 consumers with topic exchange (or do you need fanout for that)?
YES, Only if the consumer have their own (separate queue bind with same routing key).Otherwise it will be Round Robin way.
So best way is to have consumer's own queue with required routing key...!!!
For a typical topic exchange set up; how many queues do you need?
It depends on your application needs. You may start from just one queue to implement simple FIFO stack and later add more queues with for more granulated messages consuming.
Can my consumers get away with just exchange binding, or does it have to include queue interaction?
The idea of AMQP exchanges is to get published messages and put them to one or more queues (or even other exchanges or drop at all under certain condition). Consumer works only with queues, while publisher works only with exchanges.
There might be some misunderstandings with Default Exchange while it route messages to the queue name same as a routing key and it sometimes interpreted as publishing to queues which is not true.
Is it possible for a single message to appear in 2 consumers with topic exchange (or do you need fanout for that)?
I guess you are talking about duplicating one message to multiple queues (while there are some erroneous situations when you really may have single message being processed by multiple consumers).
If so - sure. You can just create multiple bindings for queue to get different messages. Here is small example:
Let you have three messages published to topic exchange with three different routing keys customer.appA.created, customer.appB.created and `customer.appC.created.
You can create separate queue to collect specific messages by binding queue with exact routing key - customer.appA.created, customer.appB.created and so on, or as you already noted, bind queue with wildcard routing key customer.*.created.
To collect messages only for appA and appB you can create queue with two bindings customer.appA.created and customer.appB.created, so you'll get two messages type in one queue.
I am using zeromq to solve a problem which involves several hundred (potentially thousands) clients request tasks to be carried out. Each client would request for a specific task to be carried out, and the result(s), when completed, whould be returned back to the client that issued that request.
These are the actors that I have identified so far, in the pattern I have come up with:
Client: this is the actor that requests a unit of work (or 'job') to be carried out
Controller: this is the actor that loadbalances the 'jobs' accross available engines
Engine: this is the actor that receives a job request from the controller and publishes the result back to the client.
I still haven't yet worked out how the engine gets the mesage back to the client. I am guessing that one way for this to be implemented using zeromq would be:
Client:
PUSH job messages on one socket to Controller SUBscribe to completed results on PUBlished by Engine, on another
socket
Controller:
PULL job messages from client on one socket PUBlish job messages to engines on another socket (clearly, this will be a forwarding device)
Engine:
SUBscribe to job messages on one socket PUBlish result to another socket
It would be most helpful if someone provide a skeleton/snippet which will show the outline of how this pattern may be implemented, using the zeromq framework.
The code snippet can be in C, C++, PHP, Python or C#
[[Edit]]
After reading up on Task Farms (as suggested by akappa). I think this problem can indeed be modelled by a Task Farm. I have modified my original actors accordingly (and changed the title too).
It would still be very useful if someone who is familiar with zeromq, can sketch out a skeleton that would show how I can use the core components to build such a framework.
There are a variety of approaches to this, and IPython.parallel includes two such implementations with ZeroMQ - one simple and pure-zmq, and another that is more elaborate, with the Controller implemented in Python.
We split the Controller into two actors:
Hub - an out-of-the-way process that sees all traffic, and keeps track of the state of the cluster, pushing results to a database, etc., notifying clients about engine connect/disconnect, etc.
Scheduler - at its core, a simple ROUTER-DEALER device that forwards requests from the client(s) to the engines, and the replies back up.
Looking at just the task-farming part of our topology:
Scheduler is a 0MQ Queue device, with a ROUTER and DEALER socket, both of which bind.
Clients have DEALER sockets, connected to the Scheduler's ROUTER
Engines have ROUTER sockets connected to the Scheduler's DEALER
Which makes use of these two properties:
DEALERS LRU load-balance requests across peers
ROUTERs use identity prefixes to send replies back to the peer that made a particular request.
A toy load-balanced task farm with pyzmq, which routes replies back up to the requesting client: https://gist.github.com/1358832
An alternative, where the results go somewhere, but not back up to the requesting client, is the Ventilator-Sink pattern in the 0MQ Guide.
This is a classical master/slave parallel pattern (also known as "Farm" or "Task Farm").
There are billion ways to implement it.
Here there is a way to implement it using MPI, maybe it can be inspirational to you for implementing it in zeromq.