[rabbitmq-discuss] RabbitMQ writes non stop to disk

[Matthew Sackman]

On Wed, Oct 26, 2011 at 04:26:38PM -0700, Raphael Simon wrote:
> Thank you for the long explanation Matthew, you had already explained this
> to me once and I have a vague understanding of how the memory allocation
> algorithm works at a high level. I don't think what we are seeing is normal
> though and can be explained by this. From what I could gather looking at the
> files written by rabbit they contain messages addressed to queues that are
> part of the "active" group. Also even if rabbit was to write every single
> message it gets through all the queues it would not result in that many
> writes. We don't have a throughput of 6000 msg/sec which is the number of
> writes iostat reports (spikes at 17,000/s). Our rate is more like 20 msg/sec
> across all queues.

For a persistent msg to a durable queue, each message will result in the
following writes:

1. The message itself. This is heavily buffered and should be done in
one write. However, I've no idea what iostat actually reports (pages
written? or syscalls to writev? or...?)

2. The fact that the msg has been published to the queue.

3. When the msg is delivered, the fact that the msg has been delivered
by the queue.

4. When the msg is ack'd, the fact that the msg has been ack'd.

2, 3 and 4 are journalled on a per queue basis. They are tiny writes (3
and 4 especially) and should again be heavily buffered, but I think we
have some timers going on that are forcing fsyncs of this journal very
often and thus may be resulting in vastly higher numbers of writes than
necessary. However, thinking about it, I don't think that existed back
in 2.4.1.

The journal involved in 2, 3 and 4 periodically gets written out which
could result in further writes for this msg in this queue.

So I would say that for each message, you're looking at a maximum of 7
"writes". Whether these writes gets coalesced or even cancelled out
entirely is difficult to say. The average should be much lower than 7
though for your use case.

Matthew