[rabbitmq-discuss] RabbitMQ writes non stop to disk

[Matthew Sackman]

Raphael,

Rabbit uses a somewhat sophisticated mechanism to allocate RAM to
queues. The allocation of RAM to queues is extremely difficult to do -
what are the criteria that you should use to decide to prioritise one
queue over another? Furthermore, if the same msg ends up in multiple
queues then that msg will exist just once in RAM but be shared by all
those queues. So it's actually nonsensical to ask how much memory a
queue is using.

The approach we adopt results in Rabbit telling all the queues on a node
to achieve the same *duration* in RAM. I.e. if that duration is, for
example, 10 seconds, then every queue should ensure that given their
individual current ingress and egress rates, the number of messages they
hold in RAM does not represent more than 10 seconds. Messages beyond
that count will be written to disk.

The effect of this strategy is that very fast queues find that an awful
lot of messages make up that duration, and very slow queues find that
very few messages make up that duration. Thus very slow queues can end
up with a "target_ram_count" of 0 or small numbers, whilst fast queues
have much higher "target_ram_counts".

This is desireable because a slow queue is far more likely to withstand
having to write everything to disk and read from disk than a fast queue
can. As I have blogged about,
( http://www.rabbitmq.com/blog/2011/09/24/sizing-your-rabbits/ ), as a
queue gets longer, its CPU-per-msg goes up. As it starts having to write
to disk, the CPU-per-msg goes up even faster. Plus disks have much lower
bandwidth than RAM. Thus a fast queue is likely to catastrophically
consume CPU and disk bandwidth if it starts getting forced out to disk.
But for a slow queue, this is absolutely fine.

Thus in the case of a slow queue receiving a lot rate of messages, what
is the problem with Rabbit deciding to write these out to disk? It's
doing it deliberately because it believes that queue can cope with being
pushed to disk (which it seems to be able to - ingress and egress rates
match), and in order to free up vital resources (RAM) for other queues
which can not cope with being sent to disk.

Rabbit works hard to avoid crowbars - sudden events where it decides to
write out millions of messages. In order to avoid this on a queue that
slowly grows and forces RAM usage to head towards the limit, Rabbit
starts writing out to disk very early on so that the transition to
fully-on-disk operation is as smooth and unnoticable as possible. Thus
even when Rabbit is some way away from its memory limit, it can still
choose to start to push msgs out to disk so to reduce the chances of
hitting the memory limit and then realising that there are millions of
msgs that need writing out causing a massive burst to the disk which'll
disrupt performance substantially.

I hope that helps explain a little more.

Matthew