[rabbitmq-discuss] One queue and several subscribers, load balancing (using ... rabbitmq + txamqp + python)

Thu Aug 13 01:47:26 BST 2009

Is it possible to load balance messages, using one queue and several  
consumers ?

So, for example, given 1 queue and 2 consumers;
	1) a publisher sends 10 messages to the queue
	2) for each consumer;
		a) receive just 1 message at time
		b)  processes it, which takes an arbitrary amount of time (say,  
between a few seconds and a few minutes)
		c)  once processed, repeat from (a)

	Thus, if 1 consumer is tens time slower than the other, it will  
process only 1 message while the other consumer will have processed  
the remaining 9 messages.

I guess this scenario is like a queue of jobs which are processed by  
the next available worker (consumer).

However,  it appears that txamqp will cache the messages, e.g. each  
consumer will always get 5 messages each.
So, to get the kind of the behavior I want, for each consumer I do;
	a) send a basic_ack(...) on the first msg, so acknowledge just the  
first message txamqp yields
	b) ignore any other messages tzamqp may have queued up
	b) send a basic_recover(True), so that rabbitmq will redistributing  
the remaining messages to all consumers

However, this method doesn't seem right to me.  I'm worried it will;
	a) create a lot of unwanted AMQP traffic
	b) will occasionally allow the one message to be received by two or  
more consumers,
		e.g. one consumer might send a basic_recover(...) just before  
another consumer is about to acknowledge it's received message.

So is there a way to only get one message at a time for a given queue  
for a given consumer ?

Any help is much appreciated.
Thanks
Dale