[rabbitmq-discuss] ETS memory usage

[Simone Sciarrati]

Hi,

Thanks a lot. This node has the most active active queue and it is
also the stats node for the cluster, I can see it is doing lots of IO
compared to the other nodes and as you say it might be lagging behind.
Is there a way to move the stats to another node in the cluster
without having to restart RabbitMQ?

Thanks,

Simone

On Mon, Oct 7, 2013 at 1:16 PM, Simon MacMullen <simon at rabbitmq.com> wrote:
> On 07/10/2013 10:41AM, Simone Sciarrati wrote:
>>
>> I am investigating memory usage on one of our nodes in a cluster of 3
>> x c1.xlarge instances in ec2, ubuntu 12.04, RabbitMQ 2.8.4 and Erlang
>> R14B04 (2 disk nodes and one ram node) . We are planning to upgrade to
>> 3.1.x but regardless of the version I would like to understand how to
>> extract information about what is consuming the memory.
>
>
> You are aware that later versions of RabbitMQ do some of this for you?
> http://www.rabbitmq.com/memory-use.html
>
>
>> Right now, one of the nodes is showing 2.3GB used (high watermark
>> 2.7GB), where 2GB are held in ETS according to the management console:
>
>
> <snip>
>
>
>> According to this output the 2 tables that are consuming most of the
>> memory are:
>>
>>   2629654         anon              ordered_set 1308748 215451711
>> <0.9548.283>
>>   471110          rabbit_msg_store_ets_index set   2627724 39819062
>> msg_store_persistent
>
>
> <snip>
>
>
>> Looking at this output the sum of the memory doesn't seem to add up to
>>>
>>> 2GB
>
>
> IIRC the output from some of the ETS statistics functions are sometimes in
> words rather than bytes.
>
> Also we reported discrepancies between the globally reported ETS memory use
> and the per-table use that I think were fixed in later versions of Erlang.
>
>
>> , also the number of objects in the rabbit_msg_store_ets_index is
>>>
>>> 2.5 millions, which doesn't seem correct when the number of messages
>>
>> in all queues on this node isn't more than 150k at any time (perhaps
>> my understanding of this number is incorrect and it doesn't relate to
>> the # of messages).
>
>
> It should do. If the server is very active it may lag behind the real state
> of affairs (i.e. a message might have been consumed but not yet removed from
> the message store index).
>
>
>> Any help in understanding where the memory might be used would be
>> greatly appreciated, this is a generic question not specifically tied
>> to the RabbitMQ version (same or different problem might present with
>> later versions at some point in time).
>
>
> The bulk of your memory is going to the mgmt stats db, mostly (if I'm
> counting correctly) on the channels table. There was a bug (fixed in 3.0.3)
> where stats on connections / channels would leak when a node that was not
> running the stats DB went down with active connections / channels. If that
> has happened a few times that would explain what you're seeing.
>
> Cheers, Simon
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal