[rabbitmq-discuss] Load balancing with multiple consumers on a single queue

[Matthias Radestock]

David.Corcoran at edftrading.com wrote:
> 
> rabbitmq-discuss-bounces at lists.rabbitmq.com wrote on 30/07/2008 09:33:19:
> 
> 
>> The round-robin nature of message deliveries is great for load
>> balancing if all our jobs require equal processing time but this is
>> not the case.. jobs can range anywhere from a few seconds to a
>> minute worth of processing.
> 
> 
> Hey Nathan,
> 
> We're having the exact same problem. We have jobs of varying lengths and we
> don't know how long each will take before we send it off. Some jobs can
> take a few ms and others 5 minutes. So we can't do different queues for
> different length jobs.

Wouldn't a simple solution be to have all consumers consuming off a 
single queue using basic.get? I.e. consumer gets job with basic.get, 
does work, sends ack, gets next job with basic.get, etc.

This does involve polling if the job queue is empty (basic.get doesn't 
block; it returns a get-empty response), but you could probably work out 
some reasonably straightforward heuristics to stop that from getting out 
of hand.


Matthias.