[rabbitmq-discuss] Dead-Lettering Queue Contents On Queue Expiration

[Alex Robson]

Hello all,

I would like to see messages in an expired queue published to the
dead-letter exchange (if one has been provided) instead of having them lost.

Our use case is somewhat complex to explain, but ultimately, setting TTL on
a message is very different from TTL on a queue in that queue expiry is
based on the presence of a consumer. The utility of having both the queue
disposed and messages re-published seems pretty high to me. Let me try an
explain our use case.

A common pattern I've seen is to have multiple instances of the same
service consuming from the same queue. This makes the system resilient to
individual service instance failures because remaining instances (or new
ones) will continue consuming messages from the queue.

A problem in this pattern is when common state is being mutated as a result
of the messages; you end up with race conditions/last-write-wins style
behaviors that can become problematic.

The way we solve for this (within the realm of Rabbit) is to have a
consistent-hash exchange that hashes the correlation-id which is set to our
aggregate root (parent record id). This allows us to bind a
queue-per-service-instance and get the isolation guarantees we like while
still allowing us to horizontally scale. The problem with this approach is
that in the event that a service loses its connection/falls over, messages
in the queue sit and can be 'orphaned'.

If we use a queue TTL policy for these kinds of queues, it will allow us to
prevent orphaned queues, but presently it does not appear that the messages
in the queue are published to the defined dead letter exchange. Since
messages with a TLL behave this way, it seems logical that messages within
a queue that expires could/should also take advantage of the same behavior.

I believe it would be a real win for users everywhere as it allows for a
really clean way to scale horizontally without increasing race-conditions,
requiring distributed locking patterns or introducing failure modes that
could result in message loss or dramatically increased latency in message
processing.

Interested in your thoughts on this. If folks like the idea but there's not
a great deal of bandwidth, I'd be happy to attempt an initial
implementation via PR.

Thanks so much for RabbitMQ, it's awesome.

Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140114/9a9c030a/attachment.html>