[rabbitmq-discuss] When are persistent messages flushed to disk?

[Iain Hull]

Hi,

I know that this is a very difficult question to answer, but under my
current testing scenario I am losing most of my messages.

I have a simple test that pushes 1,000,000 messages to the broker as
fast as possible.  Once this is finished I kill the broker.  

* When the messages are small < 100 bytes only 250,000 of the message
survive.  I can see in the Web console the number of messages in the
queue raises very slowly compared to the rate the client sends new
messages.
* When the message is larger = 1Kb all or most of the messages are
persisted.  I can see in the Web console the number of messages in the
queue closely matches the number of messages the client has sent.

I know that my test is a little unrealistic but we are planning to use
RabbitMQ to smooth out bursts of activity on our ESB caused by batch
updates, I expect most messages to arrive in large bursts and be less
than 1 kb.  I am trying to understand the nature of the window where
persistent messages could be lost.  

* Is it time based? flushed to disk every n milliseconds.
* Is it memory based? Flushed after a buffer of a certain size is
filled.  (Our messages will be small < 1k)
* Does the rate that messages arrive change the window?
* Is most of the buffering at the socket level or in side RabbitMQ?

This part of our system is best effort delivery, so we can accept some
message lose, if we can quantify the lose if/when it happens and
understand its nature like when messages could be lost and how many
could be affected.  Also are there any parameters to tune this behavior?

Sorry if this question has been answered already, I have looked through
the mailing list for the past few years and didn't see all the details I
am looking for.

Regards,
Iain.