[rabbitmq-discuss] Load balancing with multiple consumers on a single queue

[Gordon Sim]

Gordon Sim wrote:
> David.Corcoran at edftrading.com wrote:
>> rabbitmq-discuss-bounces at lists.rabbitmq.com wrote on 30/07/2008 18:02:30:
>>
>> Hey Matthias,
>>> Wouldn't a simple solution be to have all consumers consuming off a
>>> single queue using basic.get? I.e. consumer gets job with basic.get,
>>> does work, sends ack, gets next job with basic.get, etc.
>> Not really because a lot of our jobs are only a few ms so the rtt would
>> play too much part if we did it this way. The idea of having a local queue
>> full of messages works great for us, as long as there's a way to reject
>> messages if someone else's queue is empty.
> 
> Could you use basic.recover(requeue=true) in that case? You could ack 
> the first n messages on your local queue that you want to deal with, 
> then issue the recover to have any subsequently delievered messages 
> requeued.

Sorry, didn't think this one through enough! Of course you can't ack 
before you process the tasks in case the client dies before actually 
doing the work and the messages are lost.

So for this to be of any use you would have to finish processing your 
current task, ack that then requeue everything everything else. That may 
be less attractive an option if it happens often as it removes the value 
of the prefetch.

> Of course when requeued those messages will be allocated to available 
> consumers, so some may come back to the same subscriber (that would also 
> be the case if you ack and sent them back to the same queue yourself).
> 
> To avoid that (without having some form of flow control or prefetch 
> limit) you would I think need to temporarily cancel your subscription 
> and re-subscribe when you have processed the n messages (or a portion 
> thereof) from your local queue.
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>