[rabbitmq-discuss] Load balancing with multiple consumers on a single queue

[David.Corcoran at edftrading.com]

rabbitmq-discuss-bounces at lists.rabbitmq.com wrote on 30/07/2008 09:33:19:


> The round-robin nature of message deliveries is great for load
> balancing if all our jobs require equal processing time but this is
> not the case.. jobs can range anywhere from a few seconds to a
> minute worth of processing.


Hey Nathan,

We're having the exact same problem. We have jobs of varying lengths and we
don't know how long each will take before we send it off. Some jobs can
take a few ms and others 5 minutes. So we can't do different queues for
different length jobs. One way we were looking at doing it is when a
consumer has an empty queue it sends to a topic saying so. If another
consumer has multiple jobs in its queue it negative acks the extra jobs and
lets rabbitmq redistribute them more evenly. The problem with this is that
we can't find a mechanism to negative ack. Cancelling the consumer, as you
pointed out, doesn't work because you have to ack first. We can't do this
because we could lose a job that way.

The rabbitmq javadocs have classes called AMQP.Basic.Reject and
AMQP.Basic.Return that look like they might do the job but unfortunately
they are not commented so I'm not sure if they're what I'm looking for. It
would be great if they could be used to selectively Nak.

> This was done using Barry Pederson's amqplib-0.3 library in Python.

I use the java library and it looks like ChannelN.java could do a basicNak
in the same way it does a basicAck. I'm going to test this later. Perhaps
it's the same with the python library.

Dave




*********************************************************************
This communication contains confidential information, some or all of which may be privileged. It is for the intended recipient only and others must not disclose, distribute, copy, print or rely on this communication. If an addressing or transmission error has misdirected this communication, please notify the sender by replying to this e-mail and then delete the e-mail. E-mail sent to EDF Trading may be monitored by the company. Thank you. 
EDF Trading Limited
80 Victoria Street, 3rd Floor, Cardinal Place, London, SW1E 5JL
A Company registered in England No. 4255974. 
Switchboard: 020 7061 4000
EDF Trading Markets Limited is a member of the EDF Trading Limited Group and is authorised and regulated by the Financial Services Authority.
VAT number: GB 735 5479 07
*********************************************************************