[rabbitmq-discuss] ETS memory usage

Mon Oct 7 13:36:27 BST 2013

Yes, I believe so.

Cheers, Simon

On 07/10/2013 1:32PM, Simone Sciarrati wrote:
> I think I found the answer in a previous post
> (http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-April/019446.html):
>
> You should be able to do:
>
> rabbitmqctl eval 'application:stop(rabbitmq_management).'
> rabbitmqctl eval 'application:start(rabbitmq_management).'
>
> to just restart the management plugin.
>
> I assume if I just stop it another node in the cluster will take the stats.
>
> Simone
>
> On Mon, Oct 7, 2013 at 1:50 PM, Simone Sciarrati <s.sciarrati at gmail.com> wrote:
>> Hi,
>>
>> Thanks a lot. This node has the most active active queue and it is
>> also the stats node for the cluster, I can see it is doing lots of IO
>> compared to the other nodes and as you say it might be lagging behind.
>> Is there a way to move the stats to another node in the cluster
>> without having to restart RabbitMQ?
>>
>> Thanks,
>>
>> Simone
>>
>> On Mon, Oct 7, 2013 at 1:16 PM, Simon MacMullen <simon at rabbitmq.com> wrote:
>>> On 07/10/2013 10:41AM, Simone Sciarrati wrote:
>>>>
>>>> I am investigating memory usage on one of our nodes in a cluster of 3
>>>> x c1.xlarge instances in ec2, ubuntu 12.04, RabbitMQ 2.8.4 and Erlang
>>>> R14B04 (2 disk nodes and one ram node) . We are planning to upgrade to
>>>> 3.1.x but regardless of the version I would like to understand how to
>>>> extract information about what is consuming the memory.
>>>
>>>
>>> You are aware that later versions of RabbitMQ do some of this for you?
>>> http://www.rabbitmq.com/memory-use.html
>>>
>>>
>>>> Right now, one of the nodes is showing 2.3GB used (high watermark
>>>> 2.7GB), where 2GB are held in ETS according to the management console:
>>>
>>>
>>> <snip>
>>>
>>>
>>>> According to this output the 2 tables that are consuming most of the
>>>> memory are:
>>>>
>>>>    2629654         anon              ordered_set 1308748 215451711
>>>> <0.9548.283>
>>>>    471110          rabbit_msg_store_ets_index set   2627724 39819062
>>>> msg_store_persistent
>>>
>>>
>>> <snip>
>>>
>>>
>>>> Looking at this output the sum of the memory doesn't seem to add up to
>>>>>
>>>>> 2GB
>>>
>>>
>>> IIRC the output from some of the ETS statistics functions are sometimes in
>>> words rather than bytes.
>>>
>>> Also we reported discrepancies between the globally reported ETS memory use
>>> and the per-table use that I think were fixed in later versions of Erlang.
>>>
>>>
>>>> , also the number of objects in the rabbit_msg_store_ets_index is
>>>>>
>>>>> 2.5 millions, which doesn't seem correct when the number of messages
>>>>
>>>> in all queues on this node isn't more than 150k at any time (perhaps
>>>> my understanding of this number is incorrect and it doesn't relate to
>>>> the # of messages).
>>>
>>>
>>> It should do. If the server is very active it may lag behind the real state
>>> of affairs (i.e. a message might have been consumed but not yet removed from
>>> the message store index).
>>>
>>>
>>>> Any help in understanding where the memory might be used would be
>>>> greatly appreciated, this is a generic question not specifically tied
>>>> to the RabbitMQ version (same or different problem might present with
>>>> later versions at some point in time).
>>>
>>>
>>> The bulk of your memory is going to the mgmt stats db, mostly (if I'm
>>> counting correctly) on the channels table. There was a bug (fixed in 3.0.3)
>>> where stats on connections / channels would leak when a node that was not
>>> running the stats DB went down with active connections / channels. If that
>>> has happened a few times that would explain what you're seeing.
>>>
>>> Cheers, Simon
>>>
>>> --
>>> Simon MacMullen
>>> RabbitMQ, Pivotal

-- 
Simon MacMullen
RabbitMQ, Pivotal