[rabbitmq-discuss] One queue and several subscribers, load balancing (using ... rabbitmq + txamqp + python)
dale.obrien at iinet.net.au
Thu Aug 13 04:48:09 BST 2009
Thank you, that's awesome! I'll upgrade to RabbitMQ 1.6.0 and try
On 13/08/2009, at 11:47 AM, Darien Kindlund wrote:
> RabbitMQ v1.6.0 rolled out a "prefetch" (or window size?) option,
> which allows you to specify how many messages should be cached per
> consumer upon subscribe. To accomplish your goal, simply set that
> attribute to 1, when declaring and subscribing to the queue. I'm not
> familiar with txAMQP, but it should support this feature. Others more
> knowledgeable in txAMQP can provide example code; all I can tell you
> is that the feature exists on the server.
> -- Darien
> On Wed, Aug 12, 2009 at 8:47 PM, Dale<dale.obrien at iinet.net.au> wrote:
>> Is it possible to load balance messages, using one queue and several
>> consumers ?
>> So, for example, given 1 queue and 2 consumers;
>> 1) a publisher sends 10 messages to the queue
>> 2) for each consumer;
>> a) receive just 1 message at time
>> b) processes it, which takes an arbitrary amount of
>> time (say,
>> between a few seconds and a few minutes)
>> c) once processed, repeat from (a)
>> Thus, if 1 consumer is tens time slower than the other, it
>> process only 1 message while the other consumer will have processed
>> the remaining 9 messages.
>> I guess this scenario is like a queue of jobs which are processed by
>> the next available worker (consumer).
>> However, it appears that txamqp will cache the messages, e.g. each
>> consumer will always get 5 messages each.
>> So, to get the kind of the behavior I want, for each consumer I do;
>> a) send a basic_ack(...) on the first msg, so acknowledge
>> just the
>> first message txamqp yields
>> b) ignore any other messages tzamqp may have queued up
>> b) send a basic_recover(True), so that rabbitmq will
>> the remaining messages to all consumers
>> However, this method doesn't seem right to me. I'm worried it will;
>> a) create a lot of unwanted AMQP traffic
>> b) will occasionally allow the one message to be received by
>> two or
>> more consumers,
>> e.g. one consumer might send a basic_recover(...)
>> just before
>> another consumer is about to acknowledge it's received message.
>> So is there a way to only get one message at a time for a given queue
>> for a given consumer ?
>> Any help is much appreciated.
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
More information about the rabbitmq-discuss