[rabbitmq-discuss] Load balancing with multiple consumers on a single queue

Wed Jul 30 18:39:04 BST 2008

rabbitmq-discuss-bounces at lists.rabbitmq.com wrote on 30/07/2008 18:02:30:

Hey Matthias,
>
> Wouldn't a simple solution be to have all consumers consuming off a
> single queue using basic.get? I.e. consumer gets job with basic.get,
> does work, sends ack, gets next job with basic.get, etc.

Not really because a lot of our jobs are only a few ms so the rtt would
play too much part if we did it this way. The idea of having a local queue
full of messages works great for us, as long as there's a way to reject
messages if someone else's queue is empty.

>
> This does involve polling if the job queue is empty (basic.get doesn't
> block; it returns a get-empty response), but you could probably work out
> some reasonably straightforward heuristics to stop that from getting out
> of hand.
>

I tried the polling way before and it kills RabbitMQ for us. We can have
over 500 consumers all polling and with a small wait time between polls
this causes RabbitMQ to grind to a halt. A longer wait time between polls
means we aren't efficient when there are short jobs to be run.

Dave

*********************************************************************
This communication contains confidential information, some or all of which may be privileged. It is for the intended recipient only and others must not disclose, distribute, copy, print or rely on this communication. If an addressing or transmission error has misdirected this communication, please notify the sender by replying to this e-mail and then delete the e-mail. E-mail sent to EDF Trading may be monitored by the company. Thank you. 
EDF Trading Limited
80 Victoria Street, 3rd Floor, Cardinal Place, London, SW1E 5JL
A Company registered in England No. 4255974. 
Switchboard: 020 7061 4000
EDF Trading Markets Limited is a member of the EDF Trading Limited Group and is authorised and regulated by the Financial Services Authority.
VAT number: GB 735 5479 07
*********************************************************************