[rabbitmq-discuss] One queue and several subscribers, load balancing (using ... rabbitmq + txamqp + python)

Thu Aug 13 04:48:09 BST 2009

Thank you, that's awesome!  I'll upgrade to RabbitMQ 1.6.0 and try  
using "prefetch".

On 13/08/2009, at 11:47 AM, Darien Kindlund wrote:

> Dale,
> RabbitMQ v1.6.0 rolled out a "prefetch" (or window size?) option,
> which allows you to specify how many messages should be cached per
> consumer upon subscribe.  To accomplish your goal, simply set that
> attribute to 1, when declaring and subscribing to the queue.  I'm not
> familiar with txAMQP, but it should support this feature.  Others more
> knowledgeable in txAMQP can provide example code; all I can tell you
> is that the feature exists on the server.
> -- Darien
>
> On Wed, Aug 12, 2009 at 8:47 PM, Dale<dale.obrien at iinet.net.au> wrote:
>> Is it possible to load balance messages, using one queue and several
>> consumers ?
>>
>> So, for example, given 1 queue and 2 consumers;
>>        1) a publisher sends 10 messages to the queue
>>        2) for each consumer;
>>                a) receive just 1 message at time
>>                b)  processes it, which takes an arbitrary amount of  
>> time (say,
>> between a few seconds and a few minutes)
>>                c)  once processed, repeat from (a)
>>
>>        Thus, if 1 consumer is tens time slower than the other, it  
>> will
>> process only 1 message while the other consumer will have processed
>> the remaining 9 messages.
>>
>> I guess this scenario is like a queue of jobs which are processed by
>> the next available worker (consumer).
>>
>> However,  it appears that txamqp will cache the messages, e.g. each
>> consumer will always get 5 messages each.
>> So, to get the kind of the behavior I want, for each consumer I do;
>>        a) send a basic_ack(...) on the first msg, so acknowledge  
>> just the
>> first message txamqp yields
>>        b) ignore any other messages tzamqp may have queued up
>>        b) send a basic_recover(True), so that rabbitmq will  
>> redistributing
>> the remaining messages to all consumers
>>
>> However, this method doesn't seem right to me.  I'm worried it will;
>>        a) create a lot of unwanted AMQP traffic
>>        b) will occasionally allow the one message to be received by  
>> two or
>> more consumers,
>>                e.g. one consumer might send a basic_recover(...)  
>> just before
>> another consumer is about to acknowledge it's received message.
>>
>> So is there a way to only get one message at a time for a given queue
>> for a given consumer ?
>>
>> Any help is much appreciated.
>> Thanks
>> Dale
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>