[rabbitmq-discuss] RabbitMQ binary uses a ton of memory

[Jervis Liu]

Hi,

I am having a memory problem with RabbitMQ. My setup is something like
the following:

 - 1 server exclusively running RabbitMQ on Ubunto 11 (16G memory,
set_vm_memory_high_watermark=0.8), RabbitMQ 3.0.1, Erlang R15B03, no
plugin enabled.
 - non-exclusive, durable, non-autodelete queues binds to a durable
and non-autoDelete "fanout" exchange.

As soon as I have 10000+ connections, RabbitMQ starts to eat up a lof
of memory. Most of the memory usage are kind of expected except the
memory used by binary. The memory taken by
binary seems to be consistent, i.e., each connection takes about 88K
memory. The number of 88K is not affected by whether or not there is
any message delivering nor the number is affected by using durable,
non-autodelete queues or non-durable autodelete
queues.

Below are some of the memory reports:

4000 connections/queues  (1.251G used by Rabbit. 360M by binary, 90K each conn):
[{pid,1390},
 {running_applications,[{rabbit,"RabbitMQ","3.0.1"},
                        {mnesia,"MNESIA  CXC 138 12","4.7.1"},
                        {os_mon,"CPO  CXC 138 46","2.2.10"},
                        {sasl,"SASL  CXC 138 11","2.2.1"},
                        {stdlib,"ERTS  CXC 138 10","1.18.3"},
                        {kernel,"ERTS  CXC 138 10","2.15.3"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R15B03 (erts-5.9.3.1) [source] [64-bit]
[smp:16:16] [async-threads:30] [hipe] [kernel-poll:true]\n"},
 {memory,[{total,1251251016},
          {connection_procs,201979944},
          {queue_procs,58421048},
          {plugins,0},
          {other_proc,2041234},
          {mnesia,6213536},
          {mgmt_db,0},
          {msg_index,6489376},
          {other_ets,9019392},
          {binary,360259160},
          {code,12271177},
          {atom,512601},
          {other_system,594043548}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,6738516377},
 {disk_free_limit,1000000000},
 {disk_free,8423047168},
 {file_descriptors,[{total_limit,1048476},
                    {total_used,4003},
                    {sockets_limit,943626},
                    {sockets_used,4001}]},
 {processes,[{limit,1048576},{used,44138}]},
 {run_queue,0},
 {uptime,6547}]


22000 connections/queues (4.26G used by Rabbit. 1.936G used by binary)：
 {memory,[{total,4267102664},
          {connection_procs,1084229624},
          {queue_procs,406099296},
          {plugins,0},
          {other_proc,0},
          {mnesia,85761360},
          {mgmt_db,0},
          {msg_index,33168232},
          {other_ets,32735296},
          {binary,1936619736},
          {code,12271177},
          {atom,512601},
          {other_system,704048653}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,6738516377},
 {disk_free_limit,1000000000},
 {disk_free,8411611136},
 {file_descriptors,[{total_limit,1048476},
                    {total_used,42949},
                    {sockets_limit,943626},
                    {sockets_used,21977}]},
 {processes,[{limit,1048576},{used,247898}]},
 {run_queue,0},
 {uptime,2351}]


62000 connections/queues (10.813G used by RabbitMQ. 5.422G used by binary)：
 {memory,[{total,10813159784},
          {connection_procs,3353922600},
          {queue_procs,799,643,168},
          {plugins,0},
          {other_proc,0},
          {mnesia,213449152},
          {mgmt_db,0},
          {msg_index,70512576},
          {other_ets,81005552},
          {binary,5422175992},
          {code,12271177},
          {atom,512601},
          {other_system,917633974}]},
 {vm_memory_high_watermark,0.8},
 {vm_memory_limit,13477032755},
 {disk_free_limit,1000000000},
 {disk_free,8393551872},
 {file_descriptors,[{total_limit,1048476},
                    {total_used,82643},
                    {sockets_limit,943626},
                    {sockets_used,61671}]},
 {processes,[{limit,1048576},{used,686559}]},
 {run_queue,0},
 {uptime,8120}]

Please note, above reports were taken when the system is in a quite
state, ie., no messaged published, no unacked messages, no message is
ready in the queue. The doc says "Binarys is the Memory used by shared
binary data in the Erlang VM. In-memory message bodies show up here.".
However my observation is that the binary takes ton of memory even
when there is no message is being delivered in the system. I made this
sure by starting RabbitMQ from a fresh state (rabbitmqctl reset), then
going ahead to create 22000 connections, there is absolutely no
message published to the Rabbit when I printed the memory report.
binary still took about 1.936G (88K each connection).

I am wondering if this memory usage of binary is an expected behavior
(I hope it is not) or I am missing anything?  Any help is appreciated.

Thanks,
Jervis