[rabbitmq-discuss] Memory grows very fast

Tue May 14 12:22:06 BST 2013

On 14/05/13 08:13, rabbit15 wrote:
> This is my status output
>
> ./rabbitmqctl status
> Status of node rabbit at katlabucs2 ...
> [{pid,6225},
>   {running_applications,
>       [{rabbitmq_perfmon,"Embedded Rabbit Perfmon","0.0.0"},

What is that? I suggest you remove it, in order to eliminate it as a 
potential source of the memory usage.

>        {rabbitmq_tracing,"RabbitMQ message logging / tracing","3.1.0"},
>        {rabbitmq_management,"RabbitMQ Management Console","3.1.0"},

I suggest you disable these two for testing.

>   {erlang_version,
>       "Erlang R15B02 (erts-5.9.2) [source] [64-bit] [smp:24:24]
> [async-threads:30] [hipe] [kernel-poll:true]\n"},

I suggest you upgrade to R16B.

>   {memory,
>       [{total,31946238720},
>        {connection_procs,526019096},
>        {queue_procs,3060711792},
>        {plugins,1325624},
>        {other_proc,1414282513},
>        {mnesia,820584},
>        {mgmt_db,14060336},
>        {msg_index,262032},
>        {other_ets,3983176},
>        {binary,26892354848},
>        {code,18609290},
>        {atom,703377},
>        {other_system,13106052}]},

That's a lot of memory for 600k messages of 609 bytes.

Here's what I'm seeing on my system:

$ rabbitmqctl list_queues -q
amq.gen-Gy2b_JHk7bMO3qi_4aSEkg	600000
$ rabbitmqctl status -q
[{pid,27026},
  {running_applications,[{rabbit,"RabbitMQ","%%VSN%%"},
                         {mnesia,"MNESIA  CXC 138 12","4.8"},
                         {os_mon,"CPO  CXC 138 46","2.2.11"},
                         {xmerl,"XML parser","1.3.3"},
                         {sasl,"SASL  CXC 138 11","2.3.1"},
                         {stdlib,"ERTS  CXC 138 10","1.19.1"},
                         {kernel,"ERTS  CXC 138 10","2.16.1"}]},
  {os,{unix,linux}},
  {erlang_version,"Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:8:8] 
[async-threads:30] [hipe] [kernel-poll:true]\n"},
  {memory,[{total,894421208},
           {connection_procs,48240},
           {queue_procs,441492368},
           {plugins,0},
           {other_proc,9551544},
           {mnesia,62344},
           {mgmt_db,0},
           {msg_index,39688},
           {other_ets,885912},
           {binary,420690720},
           {code,16465723},
           {atom,561761},
           {other_system,4622908}]},
  {vm_memory_high_watermark,0.4},
  {vm_memory_limit,5038887731},
  {disk_free_limit,1000000000},
  {disk_free,395367772160},
  {file_descriptors,[{total_limit,924},
                     {total_used,4},
                     {sockets_limit,829},
                     {sockets_used,2}]},
  {processes,[{limit,1048576},{used,143}]},
  {run_queue,0},
  {uptime,383}]

As you can see, that's using just ~900MB of memory.

I used the MulticastMain program that ships with the Java client for 
testing:

$ runjava.sh com.rabbitmq.examples.MulticastMain -q 1 -n 2 -s 609 -C 600000

You may want to try that too, in order to ascertain whether the results 
differ from your own test program.

Regards,

Matthias.