[rabbitmq-discuss] How Should a Messaging Client Handle Errors?

[Mike Hadlow]

Hi Rob,

Your Dead Letter Exchange sounds very interesting, just what I need in fact for option 2 :)

How long before we can expect to see this in RabbitMQ? I'm thinking I'll probably have to implement a client-side DLE for the time being and then change to use your DLE once it's available. But it's great to hear that it's happening.

A very useful feature would be some way of communicating the reason for the basic.reject or basic.nack, such as a stack trace for example. If basic.nack took a reason string that could be added as a header to the message sent to the DLE, it would be very helpful.

Thanks
Mike

-----Original Message-----
From: Rob Harrop [mailto:rob.harrop at gmail.com]
Sent: 14 July 2011 12:46
To: Mike Hadlow
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] How Should a Messaging Client Handle Errors?

Mike,

The solution you outline in option 2 sounds like the Dead-Letter Exchange support I'm working on at the moment.

Using DLEs, you can declare that a queue routes all dead messages to a specified exchange for later processing. Messages are considered dead when:

* The queue containing the message is deleted
* The queue containing the message is purged
* The queue containing is deleted due to lease expiry
* The message is rejected (basic.reject or basic.nack) with requeue=false
* The TTL for the message expires

Messages are routed to DLEs using their original routing key. If you make the DLE a fanout exchange then you can easily route all dead messages to all queues bound to the DLE.

When this feature is complete, the DLE code will ensure that persistent messages are routed through the DLE before being removed from the source queue - this should give you the guarantee of safety that you are looking for.

Regards,

Rob

Mike Hadlow wrote:
> Hi All,
>
> This is a copy of a blog post I've just written
> <http://mikehadlow.blogspot.com/2011/07/easynetq-how-should-messaging-client.html>.
> I was wondering if anyone on the group had any good suggestions for
> error handling strategies?
>
> EasyNetQ <https://github.com/mikehadlow/EasyNetQ> is my simple .NET
> API for RabbitMQ.
>
> I've started thinking about the best patterns for implementing error
> handling in EasyNetQ. One of the aims of EasyNetQ is to remove as many
> infrastructure concerns from the application developer as possible.
> This means that the API should correctly handle any exceptions that
> bubble up from the application layer.
>
> One of the core requirements is that we shouldn't lose messages when
> the application throws. The question then becomes: where should the
> message, that the application was consuming when it threw, go? There
> seem to be three choices:
>
>    1. Put the failed message back on the queue it was consumed from.
>    2. Put the failed message on an error queue.
>    3. A combination of 1 and 2.
>
> *Option 1*has the benefit that it's the out-of-the-box behaviour of
> AMQP. In the case of EasyNetQ, I would simply catch any exceptions,
> log them, and just send a noAck command back to RabbitMQ. Rabbit would
> put the message at the back of the queue and then resend it when it
> got to the front.
>
> Another advantage of this technique is that it gives competing
> consumers the opportunity to process the message. If you have more
> than one consumer on a queue, Rabbit will send the messages to them in
> turn, so this is out-of-the-box.
>
> The drawback of this method is that there's the possibility of the
> queue filling up with failed messages. The consumer would just be
> cycling around throwing exceptions and any messages that it might be
> able to to consume would be slowed down by having to wait their turn
> amongst a long queue of failed messages.
>
> Another problem is that it's difficult to manually inspect the
> messages and selectively delete or retry them.
>
> *Option 2*is harder to implement. I would have to catch exceptions,
> log them and then write the message to an error queue. I would need to
> write an error queue consumer to store the messages in a database. I
> would then need to provide the user with some way to inspect the
> messages alongside the error that caused them to arrive in the error
> queue so that they could make a ignore/retry decision.
>
> I could also implement some kind of wait-and-retry function on the
> error queue, but that would also add additional complexity.
>
> It has the advantage that the original queue remains clear of failing
> messages. Failed messages and the error condition that caused the
> failure can be inspected together, and failed messages can be manually
> ignored or retried.
>
> With the failed messages sitting in a database, it would also be
> simple to create a mechanism where those messages could be replayed on
> a developer machine to aid in debugging.
>
> *A combination of 1 and 2*. I'm moving towards thinking that a
> combination of 1 & 2 might be the best strategy. When a message fails
> initially, we simply noAck it and it goes back to the queue. AMQP
> provides a Redelivered flag, so when the messages is consumed a second
> time we can be aware that it's a retry. Unfortunately there doesn't
> seem to be a retry count in AMQP, so the best we can do is allow for a
> single retry. This has the benefit that it gives a competing consumer
> a chance to process the message.
>
> After the single retry we fall back to *Option 2*. The message is
> passed to the error queue on the second failure.
>
> I would be very interested in hearing how other people have
> implemented error handling with AMQP/RabbitMQ.
>
> Many thanks
>
> Mike
>
>
> 15below Limited: Company registered in England and Wales No 3945289
> Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB,
> United Kingdom
>
> 15below Australia Pty Limited: ABN 25 132 716 379 Level 50, 120
> Collins Street, Melbourne, Victoria 3000, Australia
>
> Please think about the environment before printing this email.
>
> **********************************************************************
> ** This email and any attachments may be confidential and/or legally
> privileged and are solely for the use of the intended recipient. If
> you have received this email in error please contact the sender. Any
> views or opinions expressed within this e-mail are solely those of the
> sender, and do not necessarily represent those of 15below unless
> otherwise specifically stated. Although 15below has taken every
> reasonable precaution to ensure that any attachment to this e-mail has
> been checked for viruses, it is strongly recommended that you carry
> out your own virus check before opening any attachment, as we cannot
> accept liability for any damage sustained as a result of software virus infection.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

 15below Limited: Company registered in England and Wales No 3945289
Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom

15below Australia Pty Limited: ABN 25 132 716 379
Level 50, 120 Collins Street, Melbourne, Victoria 3000, Australia

Please think about the environment before printing this email.

************************************************************************
This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient.  If you have received this email in error please contact the sender.  Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated.  Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection.