[rabbitmq-discuss] msg_store_transient usage question
Pete Wright
pwright at rubiconproject.com
Thu Sep 26 18:15:18 BST 2013
On 09/26/2013 01:39 AM, Simon MacMullen wrote:
> On 26/09/2013 4:37AM, Peter Wright wrote:
>> i was hoping someone could help explain to me under what circumstances
>> messages are written to the msg_store_transient directory. i have been
>> witnessing this directory grow to 10's of GB - infact it spun out of
>> control at one point consuming almost 100G of space. i do have a queue
>> with quite a few messages ready for consumption, yet when i monitor
>> memory utilization for this queue it rarely spikes above 20MB.
>
> The memory use reported by the queue does not include message bodies
> (since these are shared across queues). Also it does not include
> anything that's been paged out, so if messages are paging out it can
> be quite low anyway.
>
> The decision as to whether to page messages is made globally based on
> how much memory is in use. All released versions of RabbitMQ (since
> 2.0.0 when the persister was rewritten) will start to page out when
> 20% of the way to the high watermark. By default the high watermark is
> at 40% of physical memory, so paging will start at 8% memory used.
>
> That's probably too cautious, so the next release (3.2.x) will raise
> the paging ratio from 20% to 50% (so paging will start at 20% memory
> used by default) and also make it configurable. See
> http://next.rabbitmq.com/memory.html#memsup-paging
>
> Having said that, if msg_store_transient grows to 100GB of space, I
> suspect you probably have 100GB of messages. If you don't have 100GB
> of RAM, you're going to page out...
>
> Hope this helps.
fantastic this really clarifies things on my end! your explanation
lines up pretty much with what we are seeing on our end.
cheers,
-pete
--
Pete Wright
Senior Infrastructure Engineer
the Rubicon Project
pwright at rubiconproject.com
310.309.9298
More information about the rabbitmq-discuss
mailing list