[rabbitmq-discuss] Excessive memory consumption of one server in cluster setup
maze at reik.se
Fri Aug 24 13:25:42 BST 2012
2 days ago I have upgraded our RabbitMQ cluster (2 machines running in
HA mode) from 2.8.1 to 2.8.5.
Mainly to get those OOD fixes in, since we experienced those exact issues.
The upgrade went very smooth, but at some point one of the machines
(server2) started to allocate more
and more memory (even though all queues are more or less at 0 with
almost no outstanding acks)
server 1 uses ~200Mb
server 2 (at the point where I took it down) used ~6Gb
I run a rabbitmqctl report... but it didn't give me any insights
I run a rabbitmqctl eval 'erlang:memory().' but that didn't tell me too
much more (but I will attach that at the end)
I found people with similar problems:
but that's a while back so many things might have changed since then.
Also the memory difference was
rather minimal, whereas here the difference is _very_ significant,
especially since the node with less load
has the increased memory footprint.
I can upgrade to 2.8.6 (unfortunately I upgraded just before it was
released :-(), but I only want to do that if
there is some hope that the problem is solved.
I can bring server2 back online and try to investigate what is consuming
that much memory, but my
RabbitMQ/Erlang knowledge is not good enough, therefore reaching out for
So any help would be much appreciated.
Our setup is something like the following:
2 servers exclusively running RabbitMQ on CentOS 5.x (high
- both with web-console enabled
- both defined as disk nodes
- both running RabbitMQ 2.8.5 on Erlang R15B01 (after the
upgrade, Erlang was already at R15 before)
10 queues configured with mirroring
3 queues configured (without mirroring) only on server1 with a TTL
Most consumers are connecting to server1, server2 only in case of
We get about 1k messages/sec (with peaks much higher than that) into the
system, and each message is
passed through several queues for processing.
-bash-3.2$ sbin/rabbitmqctl eval 'erlang:memory().'
More information about the rabbitmq-discuss