[rabbitmq-discuss] precondition_failed error with amqp_client for erlang

[Matthew Sackman]

Hi Max,

On Wed, Jun 29, 2011 at 06:28:59PM -0400, Max Warnock wrote:
> I've built a behavior in erlang to subscribe to a given topic exchange and
> farm out message handling.  I'm using the rabbitmq amqp_client library for
> erlang and when I put the system under heavy load I get, on occasion, the
> following error:

Could you let us know which version of Rabbit, Erlang and the Erlang
client you're using?

> =ERROR REPORT==== 29-Jun-2011::18:02:18 ===
> ** Generic server <0.1117.0> terminating
> ** Last message in was {'$gen_cast',
>                            {method,
>                                {'channel.close',406,
>                                    <<"PRECONDITION_FAILED - unknown delivery
> tag 856">>,
>                                    60,80},

That's a double-ack (probably). Sadly, the AMQP 0-9-1 spec says that
acking is not idempotent, thus it's a fault to ack the same message
multiple times...

> The server receive loop where the ack happens looks like this:
> receive
> ...
> {#'basic.deliver'{delivery_tag = Tag, routing_key = RoutingKey},
> #amqp_msg{payload = Payload}} ->
>     amqp_channel:cast(get(amqp_channel_pid), #'basic.ack'{delivery_tag =
> Tag}),
>     spawn_and_queue(spawn_handle_message, Module, RoutingKey, Payload),
>     loop(Module);
> ...
> end

...hmmm, which is so simple that I can't see how it could go wrong: if
you're not double acking then something else must be going on to make
the broker think that it's not expecting an ack for that message, hence
the error. If you're doing some sort of reject operation - either
basic.nack or basic.reject on messages and you then subsequently ack one
of those messages then that would also cause this error. There may be
other cases as well.

> The amqp_client_sup can't seem to bring back the the client either and dies
> from the retry intensity being reached.  I've done a hefty amount of
> googling and can't seem to find where things could be going wrong.  Before
> jumping into the amqp_client code I thought I'd ask the mailing list if they
> have any ideas.  The only thing I can think is that there is a race
> condition within the client library.  I will be double checking my code to
> be sure it isn't sending the ack twice, but given the simplicity of the ack
> the only way it could is if it receives the same message (with identical
> delivery tag) from the amqp_client library twice.

It could be a bug in the client library, but I'd be a little surprised
if we're managing to duplicate messages somehow - that would be a new
level of fail for us. ;) However, the fact that the entire connection
dies is alarming and almost certainly a bug: PRECONDITION_FAILED is a
soft error and should only tear down the channel, not the whole
connection. After that, all you should have to do is create a new
channel and everything else should be ok. If that's not the case please
let us know.

Best wishes,

Matthew