[rabbitmq-discuss] Queues with high memory consumption

Tue May 8 12:22:55 BST 2012

Pablo,

On 07/05/12 19:28, Pablo Molnar wrote:
> I'm trying to figure out why the memory consumption of some queues
> are  extremely high although there is almost none messages ready to consume.
> For instance this queue has only 3 messages but consumes 2.4GB of
> RAM.

That is not necessarily a problem, in particular ...

> {
>    "exclusive_consumer_tag":"",
>    "messages_ready":0,
>    "messages_unacknowledged":3,
>    "messages":3,
>    "consumers":25,
>    "memory":2586777216,
>    "slave_nodes":[
>
>    ],
>    "backing_queue_status":{
>       "q1":0,
>       "q2":0,
>       "delta":[
>          "delta",
>          "undefined",
>          0,
>          "undefined"
>       ],
>       "q3":0,
>       "q4":0,
>       "len":0,
>       "pending_acks":3,
>       "target_ram_count":0,
>       "ram_msg_count":0,
>       "ram_ack_count":0,
>       "next_seq_id":72528926,
>       "persistent_count":3,
>       "avg_ingress_rate":42.456013780009044,
>       "avg_egress_rate":42.456013780009044,
>       "avg_ack_ingress_rate":42.456013780009044,
>       "avg_ack_egress_rate":42.456013780009044
>    },  ...

...this is a slow-moving, but not idle queue. I suspect what happened is 
that at some point the queue did have lots of messages in it, hence 
requiring lots of memory. Those messages subsequently got drained, but 
the since the queue is slow moving it can take a while for Erlang's gc 
to reclaim memory.

Unless this is causing any other problems I wouldn't worry about it.

> rabbitmqctl status
> Status of node 'rabbit at i-00000140-csm' ...
> [{pid,20106},
>  {running_applications,
>      [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","2.7.1"},
>       {rabbitmq_management,"RabbitMQ Management Console","2.7.1"},
>       {rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.7.1"},
>       {webmachine,"webmachine","1.7.0-rmq2.7.1-hg"},
>       {amqp_client,"RabbitMQ AMQP Client","2.7.1"},
>       {rabbitmq_management_agent,"RabbitMQ Management Agent","2.7.1"},
>       {rabbit,"RabbitMQ","2.7.1"},
>       {os_mon,"CPO  CXC 138 46","2.2.8"},
>       {sasl,"SASL  CXC 138 11","2.2"},
>       {mochiweb,"MochiMedia Web Server","1.3-rmq2.7.1-git"},
>       {inets,"INETS  CXC 138 49","5.8"},
>       {mnesia,"MNESIA  CXC 138 12","4.6"},
>       {stdlib,"ERTS  CXC 138 10","1.18"},
>       {kernel,"ERTS  CXC 138 10","2.15"}]},
>  {os,{unix,linux}},
>  {erlang_version,
>      "Erlang R15B (erts-5.9) [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]\n"},
>  {memory,
>      [{total,4736488576},
>       {processes,3832122172},
>       {processes_used,3832066481},
>       {system,904366404},
>       {atom,561761},
>       {atom_used,551430},
>       {binary,98523592},
>       {code,15193946},
>       {ets,717445688}]},
>  {vm_memory_high_watermark,0.6999999999761628},
>  {vm_memory_limit,5873161011}]
> ...done.

You may want to upgrade to the latest Erlang (R15B01) and rabbit 
(2.8.2), though I doubt that will make a difference.

Regards,

Matthias.