[rabbitmq-discuss] Excessive memory consumption of one server in cluster setup

[Matthias Reik]

Hi,

2 days ago I have upgraded our RabbitMQ cluster (2 machines running in 
HA mode) from 2.8.1 to 2.8.5.
Mainly to get those OOD fixes in, since we experienced those exact issues.

The upgrade went very smooth, but at some point one of the machines 
(server2) started to allocate more
and more memory (even though all queues are more or less at 0 with 
almost no outstanding acks)

server 1 uses ~200Mb
server 2 (at the point where I took it down) used ~6Gb

I run a rabbitmqctl report... but it didn't give me any insights
I run a rabbitmqctl eval 'erlang:memory().' but that didn't tell me too 
much more (but I will attach that at the end)

I found people with similar problems:
http://grokbase.com/t/rabbitmq/rabbitmq-discuss/1223qcx3gg/rabbitmq-memory-usage-in-two-node-cluster 

but that's a while back so many things might have changed since then. 
Also the memory difference was
rather minimal, whereas here the difference is _very_ significant, 
especially since the node with less load
has the increased memory footprint.

I can upgrade to 2.8.6 (unfortunately I upgraded just before it was 
released :-(), but I only want to do that if
there is some hope that the problem is solved.
I can bring server2 back online and try to investigate what is consuming 
that much memory, but my
RabbitMQ/Erlang knowledge is not good enough, therefore reaching out for 
some help.

So any help would be much appreciated.

Thx
Matthias

Our setup is something like the following:
       2 servers exclusively running RabbitMQ on CentOS 5.x (high 
watermark ~22Gb),
             - both with web-console enabled
             - both defined as disk nodes
             - both running RabbitMQ 2.8.5 on Erlang R15B01 (after the 
upgrade, Erlang was already at R15 before)
     10 queues configured with mirroring
       3 queues configured (without mirroring) only on server1 with a TTL
     Most consumers are connecting to server1, server2 only in case of 
failover

We get about 1k messages/sec (with peaks much higher than that) into the 
system, and each message is
passed through several queues for processing.

-bash-3.2$ sbin/rabbitmqctl eval 'erlang:memory().'
[{total,5445584424},
  {processes,2184155418},
  {processes_used,2184122352},
  {system,3261429006},
  {atom,703377},
  {atom_used,678425},
  {binary,3216386480},
  {code,17978535},
  {ets,4142048}]
...done.