[rabbitmq-discuss] RabbitMQ writes non stop to disk

Raphael Simon raphael at rightscale.com
Thu Oct 27 00:26:38 BST 2011


Thank you for the long explanation Matthew, you had already explained this
to me once and I have a vague understanding of how the memory allocation
algorithm works at a high level. I don't think what we are seeing is normal
though and can be explained by this. From what I could gather looking at the
files written by rabbit they contain messages addressed to queues that are
part of the "active" group. Also even if rabbit was to write every single
message it gets through all the queues it would not result in that many
writes. We don't have a throughput of 6000 msg/sec which is the number of
writes iostat reports (spikes at 17,000/s). Our rate is more like 20 msg/sec
across all queues.

We'll leave it running as is but I'm worried that like in the past it will
keep getting worst until the machine simply can't handle the load.

--
Raphael.

On Wed, Oct 26, 2011 at 4:01 PM, Matthew Sackman <matthew at rabbitmq.com>wrote:

> Raphael,
>
> Rabbit uses a somewhat sophisticated mechanism to allocate RAM to
> queues. The allocation of RAM to queues is extremely difficult to do -
> what are the criteria that you should use to decide to prioritise one
> queue over another? Furthermore, if the same msg ends up in multiple
> queues then that msg will exist just once in RAM but be shared by all
> those queues. So it's actually nonsensical to ask how much memory a
> queue is using.
>
> The approach we adopt results in Rabbit telling all the queues on a node
> to achieve the same *duration* in RAM. I.e. if that duration is, for
> example, 10 seconds, then every queue should ensure that given their
> individual current ingress and egress rates, the number of messages they
> hold in RAM does not represent more than 10 seconds. Messages beyond
> that count will be written to disk.
>
> The effect of this strategy is that very fast queues find that an awful
> lot of messages make up that duration, and very slow queues find that
> very few messages make up that duration. Thus very slow queues can end
> up with a "target_ram_count" of 0 or small numbers, whilst fast queues
> have much higher "target_ram_counts".
>
> This is desireable because a slow queue is far more likely to withstand
> having to write everything to disk and read from disk than a fast queue
> can. As I have blogged about,
> ( http://www.rabbitmq.com/blog/2011/09/24/sizing-your-rabbits/ ), as a
> queue gets longer, its CPU-per-msg goes up. As it starts having to write
> to disk, the CPU-per-msg goes up even faster. Plus disks have much lower
> bandwidth than RAM. Thus a fast queue is likely to catastrophically
> consume CPU and disk bandwidth if it starts getting forced out to disk.
> But for a slow queue, this is absolutely fine.
>
> Thus in the case of a slow queue receiving a lot rate of messages, what
> is the problem with Rabbit deciding to write these out to disk? It's
> doing it deliberately because it believes that queue can cope with being
> pushed to disk (which it seems to be able to - ingress and egress rates
> match), and in order to free up vital resources (RAM) for other queues
> which can not cope with being sent to disk.
>
> Rabbit works hard to avoid crowbars - sudden events where it decides to
> write out millions of messages. In order to avoid this on a queue that
> slowly grows and forces RAM usage to head towards the limit, Rabbit
> starts writing out to disk very early on so that the transition to
> fully-on-disk operation is as smooth and unnoticable as possible. Thus
> even when Rabbit is some way away from its memory limit, it can still
> choose to start to push msgs out to disk so to reduce the chances of
> hitting the memory limit and then realising that there are millions of
> msgs that need writing out causing a massive burst to the disk which'll
> disrupt performance substantially.
>
> I hope that helps explain a little more.
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111026/49bb87c5/attachment.htm>


More information about the rabbitmq-discuss mailing list