[rabbitmq-discuss] Implement guaranteed delivery with ReturnListener and ConfirmListener

[Michael Klishin]

On 14 July 2014 at 22:02:52, ArnavJoshi (arnav.joshi at mathworks.com) wrote:
> > For testing purposes, we are checking for the confirms for messages  
> which
> are being written to undeclared/unbound/currently unavailable  
> queues. We
> observe that although the ReturnListener method throws a 312  
> (NO_ROUTE)
> error code, the ConfirmListener still calls the ‘handleAck’  
> method, and not
> the ‘handleNack’ method as we would expect.

I can see that some may expect unroutable messages
to be nack-ed but RabbitMQ cannot know whether that's the case: in some cases
it's fine for messages to not be routed anywhere. So acks verify a bunch of things:

 1. Your connection is alive
 2. Your channel is still open (no exceptions)
 3. Routed message was delivered to all queue mirrors (if applicable)

Using a nack to indicate that the message is unroutable would confuse a lot
of apps into thinking that 1) or 2) were not true.


> If this implementation is by design – could you suggest a way in  
> which we
> can implement guaranteed delivery, i.e.:
>  
> 1. Confirm that the message has not only reached the broker, but  
> has been
> processed by one or more queues.
> 2. Throw a nack, or generate an exception when the message cannot  
> be routed
> and that the message has been persisted to disk
> 3. Making blocking calls to our publishes in the ReturnListener  
> implementation when we are trying to write to such queues?

Group messages in batches if you can. Then
publish a batch, use Channel#waitForConfirms. If it returns false, schedule
the batch for re-publishing. If a return arrives, do the same.
You'll also need to have a way to identify messages, but this is generally true
with confirms.

Batching will provide good throughput with easier to use but generally slower API (#waitForConfirms vs. ConfirmListener). 

Because basic.return may arrive after #waitForConfirms returns, you may want to schedule
a "batch check" that runs in a few seconds and completely marks the batch as successfully
delivered.

> Follow up question: If a queue process being killed is the only  
> criterion
> for ConfirmListener.handleNack being called – shouldn’t an  
> undeclared/currently unavailable queue result in handleNack  
> being called.
> Our suspicion is that the message which cannot be routed to a currently  
> unavailable queue is handled like an unroutable message, which  
> is confirmed
> immediately after basic.return but is also ‘ACK’ed, which seems  
> counter-intuitive.

Can you expand on how exactly your experiment was conducted? Did you receive
an ack for a message routed to a queue whose master was unavailable? Was it the only
queue the message was routed to? 
-- 
MK  

Staff Software Engineer, Pivotal/RabbitMQ