[rabbitmq-discuss] Load balancing with multiple consumers on a single queue

[David.Corcoran at edftrading.com]

rabbitmq-discuss-bounces at lists.rabbitmq.com wrote on 30/07/2008 15:31:45:

> ... so far we have not had anybody complaining
> about the absence of basic.reject.
>
> IIRC you are the first :-(

I'm just difficult ;)

>
> There are a couple of options that we have:
>
> 1. Obviously implement this feature;
> 2. Implement message delivery credit window management, which would be
> to solution to the root cause

I'm not sure how message delivery credits work but having read through a
little it looks like it might be size oriented. So producer and consumer
have a credit of bytes and when either hits zero they wait. Unfortunately
our long running jobs are no larger than our short running ones. So our
credit window would have be based on job running length which we don't know
in advance.

What we're really looking for is a nice pattern called "work stealing".
There are lots of papers out there on it but basically it just means taking
jobs from the end of other peoples' queues when you've nothing more in your
own queue. I'm not sure of the best way to do this in the rabbitmq world.

>
> What the timelines are for either of case I cannot say at the moment,
> we will need to consider other plans and what is coming up in 0-9 SP1.

That's ok. In the very short term I think we have a way around it.
Basically receiving jobs, acking and then resending them back to the same
queue. It's not pretty but it should do the trick.

Thanks for the help,

Dave



*********************************************************************
This communication contains confidential information, some or all of which may be privileged. It is for the intended recipient only and others must not disclose, distribute, copy, print or rely on this communication. If an addressing or transmission error has misdirected this communication, please notify the sender by replying to this e-mail and then delete the e-mail. E-mail sent to EDF Trading may be monitored by the company. Thank you. 
EDF Trading Limited
80 Victoria Street, 3rd Floor, Cardinal Place, London, SW1E 5JL
A Company registered in England No. 4255974. 
Switchboard: 020 7061 4000
EDF Trading Markets Limited is a member of the EDF Trading Limited Group and is authorised and regulated by the Financial Services Authority.
VAT number: GB 735 5479 07
*********************************************************************