[rabbitmq-discuss] Consumer crash, redelivery and prefetch
kjnilsson at gmail.com
Fri Mar 14 09:42:14 GMT 2014
It is a great shame that a mature message broker such as RabbitMQ is so
lacking in sensible poison message handling (or any strategies regarding
redelivery). I raised the issue here a few months ago and was advised the
priority of planned feature [basic / deliver / 01] had been raised but
wasn't given any indication of when it might be delivered.
After that I wrote a blog entry summarising my investigations which
includes some of the suggested workarounds (such as using multiple queues),
none of which I consider adequate.
There is an argument for not designing systems where losing a message is a
big deal but I think that is a topic of a different discussion. :)
On 13 March 2014 19:31, Thomas Riccardi <riccardi at systran.fr> wrote:
> Consumer crash, redelivery and prefetch
> I'm having issues with consumer crash, redelivery and prefetch:
> The messages ack strategy I've chosen is to ack the message only after
> the consumer has finished working on the message. I need this because I
> cannot aford any lost message, even in case of consumer crash during the
> work on the message.
> For performance reason my consumer have a non 0 prefetch, so when my
> consumer is handling a message (doing some work related to the message),
> other messages are pushed to the client so that it has them already
> available when it's done working on the previous message.
> With this, when a message makes the consumer crash, the handled message
> *and* the prefetched ones are re-queued, with the flag "redelivered" ==
> I currently have no other choice than rejecting all messages that have
> the "redelivered" flag set to true, because I have no way to distinguish
> the message that cause the crash from the prefetched ones that did
> nothing wrong.
> Indeed if I retry to work on a redelivered message, the consumer will
> crash again on the poison message, in an infinite loop. However I would
> like to not reject the previously prefetched messages since they are
> probably OK.
> This reasoning also applies to transient crashes that are not
> systematically reproduced by the message.
> Ideally the solution would be to be able to distinguish the first non
> ack'd message from the other ones when the consumer crashes, or
> alternatively having two types of ack : one "I'm starting to work on
> this message", and a second one "I've finished working on this message".
> Otherwise a redelivery count would do the trick (even if it means
> working multiple times on a message that did make the consumer crash,
> which is not efficient if we know crashes happen mostly systematically).
> The redelivery count is a "planned" feature according to
> http://www.rabbitmq.com/specification.html, but when will it be really
> Having a parameter per queue like the TTL to tell a maximum redelivery
> count before dead lettering the message would be much helpful in my case
> (a RPC above rabbitmq, which is probably not a highly specific and rare
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss