[rabbitmq-discuss] How Should a Messaging Client Handle Errors?

[Rob Harrop]

Mike,

The solution you outline in option 2 sounds like the Dead-Letter 
Exchange support I'm working on at the moment.

Using DLEs, you can declare that a queue routes all dead messages to a 
specified exchange for later processing. Messages are considered dead when:

* The queue containing the message is deleted
* The queue containing the message is purged
* The queue containing is deleted due to lease expiry
* The message is rejected (basic.reject or basic.nack) with requeue=false
* The TTL for the message expires

Messages are routed to DLEs using their original routing key. If you 
make the DLE a fanout exchange then you can easily route all dead 
messages to all queues bound to the DLE.

When this feature is complete, the DLE code will ensure that persistent 
messages are routed through the DLE before being removed from the source 
queue - this should give you the guarantee of safety that you are 
looking for.

Regards,

Rob

Mike Hadlow wrote:
> Hi All,
>
> This is a copy of a blog post I’ve just written
> <http://mikehadlow.blogspot.com/2011/07/easynetq-how-should-messaging-client.html>.
> I was wondering if anyone on the group had any good suggestions for
> error handling strategies?
>
> EasyNetQ <https://github.com/mikehadlow/EasyNetQ> is my simple .NET API
> for RabbitMQ.
>
> I’ve started thinking about the best patterns for implementing error
> handling in EasyNetQ. One of the aims of EasyNetQ is to remove as many
> infrastructure concerns from the application developer as possible. This
> means that the API should correctly handle any exceptions that bubble up
> from the application layer.
>
> One of the core requirements is that we shouldn’t lose messages when the
> application throws. The question then becomes: where should the message,
> that the application was consuming when it threw, go? There seem to be
> three choices:
>
>    1. Put the failed message back on the queue it was consumed from.
>    2. Put the failed message on an error queue.
>    3. A combination of 1 and 2.
>
> *Option 1*has the benefit that it’s the out-of-the-box behaviour of
> AMQP. In the case of EasyNetQ, I would simply catch any exceptions, log
> them, and just send a noAck command back to RabbitMQ. Rabbit would put
> the message at the back of the queue and then resend it when it got to
> the front.
>
> Another advantage of this technique is that it gives competing consumers
> the opportunity to process the message. If you have more than one
> consumer on a queue, Rabbit will send the messages to them in turn, so
> this is out-of-the-box.
>
> The drawback of this method is that there’s the possibility of the queue
> filling up with failed messages. The consumer would just be cycling
> around throwing exceptions and any messages that it might be able to to
> consume would be slowed down by having to wait their turn amongst a long
> queue of failed messages.
>
> Another problem is that it’s difficult to manually inspect the messages
> and selectively delete or retry them.
>
> *Option 2*is harder to implement. I would have to catch exceptions, log
> them and then write the message to an error queue. I would need to write
> an error queue consumer to store the messages in a database. I would
> then need to provide the user with some way to inspect the messages
> alongside the error that caused them to arrive in the error queue so
> that they could make a ignore/retry decision.
>
> I could also implement some kind of wait-and-retry function on the error
> queue, but that would also add additional complexity.
>
> It has the advantage that the original queue remains clear of failing
> messages. Failed messages and the error condition that caused the
> failure can be inspected together, and failed messages can be manually
> ignored or retried.
>
> With the failed messages sitting in a database, it would also be simple
> to create a mechanism where those messages could be replayed on a
> developer machine to aid in debugging.
>
> *A combination of 1 and 2*. I’m moving towards thinking that a
> combination of 1 & 2 might be the best strategy. When a message fails
> initially, we simply noAck it and it goes back to the queue. AMQP
> provides a Redelivered flag, so when the messages is consumed a second
> time we can be aware that it’s a retry. Unfortunately there doesn’t seem
> to be a retry count in AMQP, so the best we can do is allow for a single
> retry. This has the benefit that it gives a competing consumer a chance
> to process the message.
>
> After the single retry we fall back to *Option 2*. The message is passed
> to the error queue on the second failure.
>
> I would be very interested in hearing how other people have implemented
> error handling with AMQP/RabbitMQ.
>
> Many thanks
>
> Mike
>
>
> 15below Limited: Company registered in England and Wales No 3945289
> Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB,
> United Kingdom
>
> 15below Australia Pty Limited: ABN 25 132 716 379
> Level 50, 120 Collins Street, Melbourne, Victoria 3000, Australia
>
> Please think about the environment before printing this email.
>
> ************************************************************************
> This email and any attachments may be confidential and/or legally
> privileged and are solely for the use of the intended recipient. If you
> have received this email in error please contact the sender. Any views
> or opinions expressed within this e-mail are solely those of the sender,
> and do not necessarily represent those of 15below unless otherwise
> specifically stated. Although 15below has taken every reasonable
> precaution to ensure that any attachment to this e-mail has been checked
> for viruses, it is strongly recommended that you carry out your own
> virus check before opening any attachment, as we cannot accept liability
> for any damage sustained as a result of software virus infection.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss