[rabbitmq-discuss] msg_store_transient usage question

[Simon MacMullen]

On 26/09/2013 4:37AM, Peter Wright wrote:
> i was hoping someone could help explain to me under what circumstances
> messages are written to the msg_store_transient directory.  i have been
> witnessing this directory grow to 10's of GB - infact it spun out of
> control at one point consuming almost 100G of space.  i do have a queue
> with quite a few messages ready for consumption, yet when i monitor
> memory utilization for this queue it rarely spikes above 20MB.

The memory use reported by the queue does not include message bodies 
(since these are shared across queues). Also it does not include 
anything that's been paged out, so if messages are paging out it can be 
quite low anyway.

The decision as to whether to page messages is made globally based on 
how much memory is in use. All released versions of RabbitMQ (since 
2.0.0 when the persister was rewritten) will start to page out when 20% 
of the way to the high watermark. By default the high watermark is at 
40% of physical memory, so paging will start at 8% memory used.

That's probably too cautious, so the next release (3.2.x) will raise the 
paging ratio from 20% to 50% (so paging will start at 20% memory used by 
default) and also make it configurable. See 
http://next.rabbitmq.com/memory.html#memsup-paging

Having said that, if msg_store_transient grows to 100GB of space, I 
suspect you probably have 100GB of messages. If you don't have 100GB of 
RAM, you're going to page out...

Hope this helps.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal