[rabbitmq-discuss] How Should a Messaging Client Handle Errors?

Thu Jul 14 11:51:59 BST 2011

Hi All,

This is a copy of a blog post I've just written<http://mikehadlow.blogspot.com/2011/07/easynetq-how-should-messaging-client.html>. I was wondering if anyone on the group had any good suggestions for error handling strategies?

EasyNetQ<https://github.com/mikehadlow/EasyNetQ> is my simple .NET API for RabbitMQ.

I've started thinking about the best patterns for implementing error handling in EasyNetQ. One of the aims of EasyNetQ is to remove as many infrastructure concerns from the application developer as possible. This means that the API should correctly handle any exceptions that bubble up from the application layer.

One of the core requirements is that we shouldn't lose messages when the application throws. The question then becomes: where should the message, that the application was consuming when it threw, go? There seem to be three choices:

 1.  Put the failed message back on the queue it was consumed from.
 2.  Put the failed message on an error queue.
 3.  A combination of 1 and 2.

Option 1 has the benefit that it's the out-of-the-box behaviour of AMQP. In the case of EasyNetQ, I would simply catch any exceptions, log them, and just send a noAck command back to RabbitMQ. Rabbit would put the message at the back of the queue and then resend it when it got to the front.

Another advantage of this technique is that it gives competing consumers the opportunity to process the message. If you have more than one consumer on a queue, Rabbit will send the messages to them in turn, so this is out-of-the-box.

The drawback of this method is that there's the possibility of the queue filling up with failed messages. The consumer would just be cycling around throwing exceptions and any messages that it might be able to to consume would be slowed down by having to wait their turn amongst a long queue of failed messages.

Another problem is that it's difficult to manually inspect the messages and selectively delete or retry them.

Option 2 is harder to implement. I would have to catch exceptions, log them and then write the message to an error queue. I would need to write an error queue consumer to store the messages in a database. I would then need to provide the user with some way to inspect the messages alongside the error that caused them to arrive in the error queue so that they could make a ignore/retry decision.

I could also implement some kind of wait-and-retry function on the error queue, but that would also add additional complexity.

It has the advantage that the original queue remains clear of failing messages. Failed messages and the error condition that caused the failure can be inspected together, and failed messages can be manually ignored or retried.

With the failed messages sitting in a database, it would also be simple to create a mechanism where those messages could be replayed on a developer machine to aid in debugging.

A combination of 1 and 2. I'm moving towards thinking that a combination of 1 & 2 might be the best strategy. When a message fails initially, we simply noAck it and it goes back to the queue. AMQP provides a Redelivered flag, so when the messages is consumed a second time we can be aware that it's a retry. Unfortunately there doesn't seem to be a retry count in AMQP, so the best we can do is allow for a single retry. This has the benefit that it gives a competing consumer a chance to process the message.

After the single retry we fall back to Option 2. The message is passed to the error queue on the second failure.

I would be very interested in hearing how other people have implemented error handling with AMQP/RabbitMQ.

Many thanks

Mike

15below Limited: Company registered in England and Wales No 3945289
Registered Office: Lyndean House, 43-46 Queens Road, Brighton BN1 3XB, United Kingdom

15below Australia Pty Limited: ABN 25 132 716 379
Level 50, 120 Collins Street, Melbourne, Victoria 3000, Australia

Please think about the environment before printing this email.

************************************************************************
This email and any attachments may be confidential and/or legally privileged and are solely for the use of the intended recipient. If you have received this email in error please contact the sender. Any views or opinions expressed within this e-mail are solely those of the sender, and do not necessarily represent those of 15below unless otherwise specifically stated. Although 15below has taken every reasonable precaution to ensure that any attachment to this e-mail has been checked for viruses, it is strongly recommended that you carry out your own virus check before opening any attachment, as we cannot accept liability for any damage sustained as a result of software virus infection.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110714/8d8c642e/attachment.htm>