[rabbitmq-discuss] ETS memory usage

[Simon MacMullen]

On 07/10/2013 10:41AM, Simone Sciarrati wrote:
> I am investigating memory usage on one of our nodes in a cluster of 3
> x c1.xlarge instances in ec2, ubuntu 12.04, RabbitMQ 2.8.4 and Erlang
> R14B04 (2 disk nodes and one ram node) . We are planning to upgrade to
> 3.1.x but regardless of the version I would like to understand how to
> extract information about what is consuming the memory.

You are aware that later versions of RabbitMQ do some of this for you? 
http://www.rabbitmq.com/memory-use.html

> Right now, one of the nodes is showing 2.3GB used (high watermark
> 2.7GB), where 2GB are held in ETS according to the management console:

<snip>

> According to this output the 2 tables that are consuming most of the memory are:
>
>   2629654         anon              ordered_set 1308748 215451711 <0.9548.283>
>   471110          rabbit_msg_store_ets_index set   2627724 39819062
> msg_store_persistent

<snip>

> Looking at this output the sum of the memory doesn't seem to add up to
>> 2GB

IIRC the output from some of the ETS statistics functions are sometimes 
in words rather than bytes.

Also we reported discrepancies between the globally reported ETS memory 
use and the per-table use that I think were fixed in later versions of 
Erlang.

>, also the number of objects in the rabbit_msg_store_ets_index is
>> 2.5 millions, which doesn't seem correct when the number of messages
> in all queues on this node isn't more than 150k at any time (perhaps
> my understanding of this number is incorrect and it doesn't relate to
> the # of messages).

It should do. If the server is very active it may lag behind the real 
state of affairs (i.e. a message might have been consumed but not yet 
removed from the message store index).

> Any help in understanding where the memory might be used would be
> greatly appreciated, this is a generic question not specifically tied
> to the RabbitMQ version (same or different problem might present with
> later versions at some point in time).

The bulk of your memory is going to the mgmt stats db, mostly (if I'm 
counting correctly) on the channels table. There was a bug (fixed in 
3.0.3) where stats on connections / channels would leak when a node that 
was not running the stats DB went down with active connections / 
channels. If that has happened a few times that would explain what 
you're seeing.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal