Hi,
<br>
<br>2 days ago I have upgraded our RabbitMQ cluster (2 machines running in
HA mode) from 2.8.1 to 2.8.5.
<br>Mainly to get those OOD fixes in, since we experienced those exact issues.
<br>
<br>The upgrade went very smooth, but at some point one of the machines
(server2) started to allocate more
<br>and more memory (even though all queues are more or less at 0 with
almost no outstanding acks)
<br>
<br>server 1 uses ~200Mb
<br>server 2 (at the point where I took it down) used ~6Gb
<br>
<br>I run a rabbitmqctl report... but it didn't give me any insights
<br>I run a rabbitmqctl eval 'erlang:memory().' but that didn't tell me too
much more (but I will attach that at the end)
<br>
<br>I found people with similar problems:
<br><a class="moz-txt-link-freetext" href="http://grokbase.com/t/rabbitmq/rabbitmq-discuss/1223qcx3gg/rabbitmq-memory-usage-in-two-node-cluster">http://grokbase.com/t/rabbitmq/rabbitmq-discuss/1223qcx3gg/rabbitmq-memory-usage-in-two-node-cluster</a>
<br>but that's a while back so many things might have changed since then.
Also the memory difference was
<br>rather minimal, whereas here the difference is <span class="moz-txt-underscore"><span class="moz-txt-tag">_</span>very<span class="moz-txt-tag">_</span></span> significant,
especially since the node with less load
<br>has the increased memory footprint.
<br>
<br>I can upgrade to 2.8.6 (unfortunately I upgraded just before it was
released :-(), but I only want to do that if
<br>there is some hope that the problem is solved.
<br>I can bring server2 back online and try to investigate what is consuming
that much memory, but my
<br>RabbitMQ/Erlang knowledge is not good enough, therefore reaching out for
some help.
<br>
<br>So any help would be much appreciated.
<br>
<br>Thx
<br>Matthias
<br>
<br>Our setup is something like the following:
<br> 2 servers exclusively running RabbitMQ on CentOS 5.x (high
watermark ~22Gb),
<br> - both with web-console enabled
<br> - both defined as disk nodes
<br> - both running RabbitMQ 2.8.5 on Erlang R15B01 (after the
upgrade, Erlang was already at R15 before)
<br> 10 queues configured with mirroring
<br> 3 queues configured (without mirroring) only on server1 with a TTL
<br> Most consumers are connecting to server1, server2 only in case of
failover
<br>
<br>We get about 1k messages/sec (with peaks much higher than that) into the
system, and each message is
<br>passed through several queues for processing.
<br>
<br>-bash-3.2$ sbin/rabbitmqctl eval 'erlang:memory().'
<br>[{total,5445584424},
<br> {processes,2184155418},
<br> {processes_used,2184122352},
<br> {system,3261429006},
<br> {atom,703377},
<br> {atom_used,678425},
<br> {binary,3216386480},
<br> {code,17978535},
<br> {ets,4142048}]
<br>...done.<br><br>PS: Sorry for the eventual spam (caused by mail-box mix up :-(<br>