[rabbitmq-discuss] One queue and several subscribers, load balancing (using ... rabbitmq + txamqp + python)

[Darien Kindlund]

Dale,
RabbitMQ v1.6.0 rolled out a "prefetch" (or window size?) option,
which allows you to specify how many messages should be cached per
consumer upon subscribe.  To accomplish your goal, simply set that
attribute to 1, when declaring and subscribing to the queue.  I'm not
familiar with txAMQP, but it should support this feature.  Others more
knowledgeable in txAMQP can provide example code; all I can tell you
is that the feature exists on the server.
-- Darien

On Wed, Aug 12, 2009 at 8:47 PM, Dale<dale.obrien at iinet.net.au> wrote:
> Is it possible to load balance messages, using one queue and several
> consumers ?
>
> So, for example, given 1 queue and 2 consumers;
>        1) a publisher sends 10 messages to the queue
>        2) for each consumer;
>                a) receive just 1 message at time
>                b)  processes it, which takes an arbitrary amount of time (say,
> between a few seconds and a few minutes)
>                c)  once processed, repeat from (a)
>
>        Thus, if 1 consumer is tens time slower than the other, it will
> process only 1 message while the other consumer will have processed
> the remaining 9 messages.
>
> I guess this scenario is like a queue of jobs which are processed by
> the next available worker (consumer).
>
> However,  it appears that txamqp will cache the messages, e.g. each
> consumer will always get 5 messages each.
> So, to get the kind of the behavior I want, for each consumer I do;
>        a) send a basic_ack(...) on the first msg, so acknowledge just the
> first message txamqp yields
>        b) ignore any other messages tzamqp may have queued up
>        b) send a basic_recover(True), so that rabbitmq will redistributing
> the remaining messages to all consumers
>
> However, this method doesn't seem right to me.  I'm worried it will;
>        a) create a lot of unwanted AMQP traffic
>        b) will occasionally allow the one message to be received by two or
> more consumers,
>                e.g. one consumer might send a basic_recover(...) just before
> another consumer is about to acknowledge it's received message.
>
> So is there a way to only get one message at a time for a given queue
> for a given consumer ?
>
> Any help is much appreciated.
> Thanks
> Dale
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>