[rabbitmq-discuss] Lose messages between restarts of broker

[Matthew Sackman]

On Wed, Oct 27, 2010 at 05:15:31PM -0500, Dave Gorman wrote:
> I then publish 10,000 messages and I've set the exchange to durable, the
> queue to durable, and the message delivery to "2" - as I understand this to
> be the configuration for messages to be durable. I then use the rabbitmqctl
> list_queues to show the 10,000 messages. I close the broker window, to mimic
> a hard shutdown of a server. When I start the broker back up and list the
> queues I see 8,428 messages.
> 
> Do you know why I would lose messages?

Yes. The settings you've described result in Rabbit taking a "best
effort" approach to writing messages to disk. Rabbit will aggressively
buffer pending messages in RAM and may indeed never write some such
messages to disk.

If you really need messages to go safely to disk, then you should put
the channel into transactional mode (tx.select), and issue a tx.commit
when you have published your 10000 messages. When you get the
tx.commit-ok message back, it is guaranteed the messages are safely
written to disk (persistent messages sent to a durable queue).

Obviously, the frequency with which you issue the tx.commit determines
the number of messages that can be lost: if you do a tx.commit after
every single publish then you'll lose at most one message (death during
publish/commit). Doing that, whilst very safe, has a substantial impact
on performance.

Best wishes,

Matthew