[rabbitmq-discuss] ETS memory usage

[Simone Sciarrati]

Hi,

For the record, that worked as expected, the memory usage dropped
1.3GB to normal (around 1G) and another node in the cluster became the
stats node. The command took some time to complete (2/3 minutes) but
there was no interruption of service.

Simone

On Mon, Oct 7, 2013 at 2:36 PM, Simon MacMullen <simon at rabbitmq.com> wrote:
> Yes, I believe so.
>
> Cheers, Simon
>
>
> On 07/10/2013 1:32PM, Simone Sciarrati wrote:
>>
>> I think I found the answer in a previous post
>>
>> (http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-April/019446.html):
>>
>> You should be able to do:
>>
>> rabbitmqctl eval 'application:stop(rabbitmq_management).'
>> rabbitmqctl eval 'application:start(rabbitmq_management).'
>>
>> to just restart the management plugin.
>>
>> I assume if I just stop it another node in the cluster will take the
>> stats.
>>
>> Simone
>>
>> On Mon, Oct 7, 2013 at 1:50 PM, Simone Sciarrati <s.sciarrati at gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> Thanks a lot. This node has the most active active queue and it is
>>> also the stats node for the cluster, I can see it is doing lots of IO
>>> compared to the other nodes and as you say it might be lagging behind.
>>> Is there a way to move the stats to another node in the cluster
>>> without having to restart RabbitMQ?
>>>
>>> Thanks,
>>>
>>> Simone
>>>
>>> On Mon, Oct 7, 2013 at 1:16 PM, Simon MacMullen <simon at rabbitmq.com>
>>> wrote:
>>>>
>>>> On 07/10/2013 10:41AM, Simone Sciarrati wrote:
>>>>>
>>>>>
>>>>> I am investigating memory usage on one of our nodes in a cluster of 3
>>>>> x c1.xlarge instances in ec2, ubuntu 12.04, RabbitMQ 2.8.4 and Erlang
>>>>> R14B04 (2 disk nodes and one ram node) . We are planning to upgrade to
>>>>> 3.1.x but regardless of the version I would like to understand how to
>>>>> extract information about what is consuming the memory.
>>>>
>>>>
>>>>
>>>> You are aware that later versions of RabbitMQ do some of this for you?
>>>> http://www.rabbitmq.com/memory-use.html
>>>>
>>>>
>>>>> Right now, one of the nodes is showing 2.3GB used (high watermark
>>>>> 2.7GB), where 2GB are held in ETS according to the management console:
>>>>
>>>>
>>>>
>>>> <snip>
>>>>
>>>>
>>>>> According to this output the 2 tables that are consuming most of the
>>>>> memory are:
>>>>>
>>>>>    2629654         anon              ordered_set 1308748 215451711
>>>>> <0.9548.283>
>>>>>    471110          rabbit_msg_store_ets_index set   2627724 39819062
>>>>> msg_store_persistent
>>>>
>>>>
>>>>
>>>> <snip>
>>>>
>>>>
>>>>> Looking at this output the sum of the memory doesn't seem to add up to
>>>>>>
>>>>>>
>>>>>> 2GB
>>>>
>>>>
>>>>
>>>> IIRC the output from some of the ETS statistics functions are sometimes
>>>> in
>>>> words rather than bytes.
>>>>
>>>> Also we reported discrepancies between the globally reported ETS memory
>>>> use
>>>> and the per-table use that I think were fixed in later versions of
>>>> Erlang.
>>>>
>>>>
>>>>> , also the number of objects in the rabbit_msg_store_ets_index is
>>>>>>
>>>>>>
>>>>>> 2.5 millions, which doesn't seem correct when the number of messages
>>>>>
>>>>>
>>>>> in all queues on this node isn't more than 150k at any time (perhaps
>>>>> my understanding of this number is incorrect and it doesn't relate to
>>>>> the # of messages).
>>>>
>>>>
>>>>
>>>> It should do. If the server is very active it may lag behind the real
>>>> state
>>>> of affairs (i.e. a message might have been consumed but not yet removed
>>>> from
>>>> the message store index).
>>>>
>>>>
>>>>> Any help in understanding where the memory might be used would be
>>>>> greatly appreciated, this is a generic question not specifically tied
>>>>> to the RabbitMQ version (same or different problem might present with
>>>>> later versions at some point in time).
>>>>
>>>>
>>>>
>>>> The bulk of your memory is going to the mgmt stats db, mostly (if I'm
>>>> counting correctly) on the channels table. There was a bug (fixed in
>>>> 3.0.3)
>>>> where stats on connections / channels would leak when a node that was
>>>> not
>>>> running the stats DB went down with active connections / channels. If
>>>> that
>>>> has happened a few times that would explain what you're seeing.
>>>>
>>>> Cheers, Simon
>>>>
>>>> --
>>>> Simon MacMullen
>>>> RabbitMQ, Pivotal
>
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal