[rabbitmq-discuss] Incorrect number of messages saved to the queue after RabbitMQ broker failure

[Matthew Sackman]

On Tue, Mar 08, 2011 at 04:16:48PM -0800, venkat veludandi wrote:
> After consuming few published messages, if I bring down the RabbitMQ broker (hard failure) and bring it back, the number of messages saved to queue is incorrect.
> For example, I publish 100 messages. After consuming 10 messages, I bring down the broker and bring it back. Run: rabbitmqctl list_queues messages_ready, I see the count of messages_ready as 100 instead of 90. 
> Could you please let me know if there is anything wrong with the following Producer and consumer code:

Ok, so the publishing side of things is fine: by the time you get the
tx.commit_ok back, the server has correctly written the messages to
disk.

On the acking side, because you're not using transactions, there is no
guarantee given as to when the ack is written to disk. Consequently,
as your test shows, ack'd messages can come back to life in the event of
a hard kill. However, in this case you will find that messages you
subsequently consume upon recovery will have the is_delivered property
set (indeed, it will in fact be set for all messages recovered by the
queue as the queue, in the case of a hard kill, has to be conservative
and assume that all messages in the queue _may_ have been previously
delivered).

To eliminate this problem, you can issue acks within a transaction.
Then, once again, by the time you get the tx.commit_ok back, you know
the ack has been recorded.

On the whole, we do not advise people to try and seek exactly-once
semantics. It is very difficult if not impossible to get anywhere near
exactly-once - this isn't just with RabbitMQ: this is a standard
theoretical result. You can easily have "at least once" and "at most
once" delivery. The former is generally preferred as it means you won't
lose messages, but your consumers have to deal with the possibility of
duplication. This is usually possible.

Thus once you've embraced this, you will find performance is several
orders of magnitude better if you use confirms when publishing[0], and
normal acks when consuming. At that point there is likely no need to use
transactions at all. Try to ensure your downstream operations are
idempotent and thus can naturally cope with duplication of the
occasional message.

Matthew

[0] http://www.rabbitmq.com/extensions.html#confirms