[rabbitmq-discuss] Load balancing with multiple consumers on a single queue

[Tony Garnock-Jones]

Hi Nathan,

Nathan Oorloff wrote:
> the client library gets a basic_deliver 
> instead of the basic_cancel_ok it expected.

Yep: this is because until you hear the cancel-ok from the server, you 
can't assume the server has heard the cancel request yet. The delivery 
could have already been in flight at the time the cancel was sent by the 
client.

There are only a very few kinds of "method" than can appear unsolicited 
from the server on a fully-established channel: channel.close, 
basic.return and basic.deliver. A client library has to be prepared to 
receive these at any time, even when it's waiting for a reply to a 
synchronous method request it sent earlier.

> I edited the library to not 
> error in this instance and to ignore messages after the cancel had been 
> sent

Better would be to deliver messages to the application until the 
cancel-ok is received. The server does consider these to be legitimate 
deliveries, and it is waiting (if noack wasn't enabled) for your 
acknowledgement of them.

> however when I start the consume again I have a client waiting for 
> a message to finish getting delivered it seems which never happens. 

I don't understand what you mean here.

> Tonyg (on irc) said for the cancel to work properly I'd probably have to 
> cancel the channel/connection rather than just doing a basic_cancel. 

If by "properly", you mean requeueing any unacknowledged deliveries on 
the channel, so that they can be redelivered to the same or a different 
channel, yes (another options is to use basic.recover).

Cancelling a consumer does not affect any of the deliveries that were 
sent your way during the lifetime of the consumer: in order to let the 
server know that you don't want them, you need to reject, recover, or 
close the channel.

> I might give "recover" a test. At first glance it looks like I should 
> run it after I've ack'd the message I'm about to process and issued the 
> cancel?

"recover" is not very well defined in 0-8, because it lacks a 
corresponding "recover-ok" which would tell the issuing client where the 
boundary between deliveries pre-recover and deliveries post-recover 
sits. To work around this problem, I would

  - acknowledge any of the deliveries I've heard about that I want
    to keep (this step can be done at any time prior to the sending of
    the basic.recover command)

  - cancel ALL consumers on the channel concerned, and wait for the
    corresponding cancel-ok messages to arrive. This gives a guarantee
    that no more basic.delivers will appear before you issue the recover,
    so you know exactly what's going on.

  - in the client, discard any record of any outstanding unprocessed,
    unacknowledged deliveries: essentially, give yourself a clean slate

  - issue the basic.recover

  - start consumers at your leisure

Painful. Then again, basic.recover does seem like kind of an unusual 
thing to want to do -- perhaps those cases where its use is required are 
"bad code smells" indicating weaknesses in the main part of the AMQP 
protocol?

By the way, thanks to all the participants in this thread so far. It's 
exactly these kinds of use-cases and discussions that will help make 
AMQP (and its implementations) rock-solid.

Regards,
   Tony