[rabbitmq-discuss] Acked Message getting redelivered

[Iain Hull]

Hi,

 

We are using RabbitMQ 2.5.1 with the java client.  Our application
requires at quite reliable at most once delivery QoS Semantics. We
achieve this by using persistent messages and calling basic_ack just
before processing each message (using the channel that delivered the
message).  If the ack fails we discard the message and try the next
message from the consumers internal in memory queue.  If a channel or
connection is closed, a management thread re-establishes the channel or
connection and reconnects the consumer asynchronously, so all unacked
messages get redelivered to the consumer.  The original version of these
messages might still be in the consumers internal queue, but since we
try to ack them before processing they will be silently ignored.

 

The problem is some messages are getting processed twice.  This means
that a message that is acked successfully by the client is getting
redelivered when the connection drops at the same time as the ack.  We
currently have some network conditions that we haven't figured out yet
that cause the connection to drop every 15 minutes.  When this happens
the message sometimes gets redelivered to consumer on a different
system.

 

I have attached a network diagram to explain our topology.  We have two
rabbitmq boxed in a cluster, but they are not involved in any HA or
automatic failover.  Failover is handled by the clients on the producer
and consumer systems.  The queues and exchanges are duplicated across
each server (with a server identifier added).  We have multiple producer
systems each connecting to one rabbitmq server (chosen at random).
There are two ESB servers forming a cluster that consume the messages
from the queues on both rabbitmq servers.  The consumers on each ESB
merge the queues from each rabbitmq server and present it as one queue
to the application layer.  The application layer does not get the
message until after it is acked. 

 

Obviously we have to fix our network issues.  However we would also like
to understand how we can process messages twice when a connection drops.
Is the ack asynchronous? Or is there a chance of a race condition when a
connection is dropped during an ack where the client considers the
message acked but the server does not?  Or does this look like a bug?

 

I haven't yet made a small application to reproduce this but should be
able to if required.

 

Regards,

Iain.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120118/18dbeb3c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Rabbit_Topology.png
Type: image/png
Size: 19483 bytes
Desc: Rabbit_Topology.png
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120118/18dbeb3c/attachment.png>