[rabbitmq-discuss] One queue and several subscribers, load balancing (using ... rabbitmq + txamqp + python)
dale.obrien at iinet.net.au
Thu Aug 13 01:47:26 BST 2009
Is it possible to load balance messages, using one queue and several
So, for example, given 1 queue and 2 consumers;
1) a publisher sends 10 messages to the queue
2) for each consumer;
a) receive just 1 message at time
b) processes it, which takes an arbitrary amount of time (say,
between a few seconds and a few minutes)
c) once processed, repeat from (a)
Thus, if 1 consumer is tens time slower than the other, it will
process only 1 message while the other consumer will have processed
the remaining 9 messages.
I guess this scenario is like a queue of jobs which are processed by
the next available worker (consumer).
However, it appears that txamqp will cache the messages, e.g. each
consumer will always get 5 messages each.
So, to get the kind of the behavior I want, for each consumer I do;
a) send a basic_ack(...) on the first msg, so acknowledge just the
first message txamqp yields
b) ignore any other messages tzamqp may have queued up
b) send a basic_recover(True), so that rabbitmq will redistributing
the remaining messages to all consumers
However, this method doesn't seem right to me. I'm worried it will;
a) create a lot of unwanted AMQP traffic
b) will occasionally allow the one message to be received by two or
e.g. one consumer might send a basic_recover(...) just before
another consumer is about to acknowledge it's received message.
So is there a way to only get one message at a time for a given queue
for a given consumer ?
Any help is much appreciated.
More information about the rabbitmq-discuss