[rabbitmq-discuss] Dead-Lettering Queue Contents On Queue Expiration

[Matthias Radestock]

Alex,

On 17/01/14 22:08, Alex Robson wrote:
> I have updated the pull request. I jumped the gun a bit in thinking that
> I had solved this with the initial pull. I still may not have resolved
> it in a way that would make sense to devs who know the code base better
> than I do. I wasn't sure if there were a better way to remove bindings
> from an expiring queue and its DLX, so I'm removing them first from the
> destination using rabbit_binding:remove_for_destination and then calling
> the exchanges remove_bindings. The latter call is necessary because with
> out it, a consistent hash exchange will retain invalid routes and
> continue to try and route messages to the expired queue.

I can see why this is necessary, but it makes a whole bunch of 
assumptions about topology that work for c-h-x but not in general.

Wwhy not remove all bindings, indeed even perhaps the queue itself? i.e. 
bring forward part of what happens in terminate_delete, in particular 
the call to rabbit_amqqueue:internal_delete.

That is not w/o problems either since now you have a "dark" queue, i.e. 
a queue that is completely invisible to routing and management. Yet that 
queue is still operating - it's dead-lettering - and using the backing 
queue, etc. That is likely problematic. So, for example, a client could 
come along and create a new queue with the same name, which would end up 
stomping on or tripping over the persistent storage of the dark queue.

Yet if you don't remove the queue record then there is nothing stopping 
clients from creating new bindings for it. Including to the DLX.

So perhaps this queue record in mnesia needs to indicate some limbo 
state, similar to what happens to durable queues residing on down nodes. 
In that state, the existence of the queue is known but no operations can 
be performed on it.

This is all getting rather tricky.

Plus we still have to deal with the issue I mentioned previously, of 
messages that are in-flight between the exchange and 
in-the-process-of-deletion queue getting lost. That can actually be 
addressed for specific use cases, namely those where the message is 
expected to reach exactly one queue - by publishing in mandatory mode 
and re-sending the message when a basic.return is received. Obviously 
this can create dups, but they are inevitable in most cases anyway. It's 
also rather slow since it's synchronous (though that will change in the 
next RabbitMQ feature release).

Trouble is, you'd also have to get the dead-lettering to publish in 
mandatory mode. Which introduces all sorts of complications.

Also note, btw, that, in general, dead-lettering is not guaranteed. It 
doesn't use confirms (and doing so would be very complex indeed), so 
messages can get lost due to failures during dead lettering.


Summary: It's all rather complicated.


Regards,

Matthias.