Just to make sure, I stopped the application, reset it, and rejoined the cluster (on server2).<br>Thus, any "old" information in mnesia that could have caused that strange behavior should <br>be wiped out.<br><br>
Cheers<br>Matthias<br><br><div class="gmail_quote">On Mon, Aug 27, 2012 at 11:26 AM, Matthias Reik <span dir="ltr"><<a href="mailto:matthias.reik@gmail.com" target="_blank">matthias.reik@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I just upgraded to 2.8.6, and I see the same effect with the latest version :-(<br>(not really unexpected, since nothing was fixed in that regard in 2.8.6).<br>
<br>Already after about 10 minutes server2 consumes 200% more memory than server1, <br>
even though all clients are connected to server1, and there are no queues with TTL anymore.<br><br>(removing the other queues is unfortunately not possible, since it's a production<br>server :-(, and I haven't seen this effect on any of our other (development) servers)<br>
<br>Cheers<span class="HOEnZb"><font color="#888888"><br>Matthias</font></span><div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">On Fri, Aug 24, 2012 at 2:25 PM, Matthias Reik <span dir="ltr"><<a href="mailto:maze@reik.se" target="_blank">maze@reik.se</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
2 days ago I have upgraded our RabbitMQ cluster (2 machines running in HA mode) from 2.8.1 to 2.8.5.<br>
Mainly to get those OOD fixes in, since we experienced those exact issues.<br>
<br>
The upgrade went very smooth, but at some point one of the machines (server2) started to allocate more<br>
and more memory (even though all queues are more or less at 0 with almost no outstanding acks)<br>
<br>
server 1 uses ~200Mb<br>
server 2 (at the point where I took it down) used ~6Gb<br>
<br>
I run a rabbitmqctl report... but it didn't give me any insights<br>
I run a rabbitmqctl eval 'erlang:memory().' but that didn't tell me too much more (but I will attach that at the end)<br>
<br>
I found people with similar problems:<br>
<a href="http://grokbase.com/t/rabbitmq/rabbitmq-discuss/1223qcx3gg/rabbitmq-memory-usage-in-two-node-cluster" target="_blank">http://grokbase.com/t/<u></u>rabbitmq/rabbitmq-discuss/<u></u>1223qcx3gg/rabbitmq-memory-<u></u>usage-in-two-node-cluster</a> <br>
but that's a while back so many things might have changed since then. Also the memory difference was<br>
rather minimal, whereas here the difference is _very_ significant, especially since the node with less load<br>
has the increased memory footprint.<br>
<br>
I can upgrade to 2.8.6 (unfortunately I upgraded just before it was released :-(), but I only want to do that if<br>
there is some hope that the problem is solved.<br>
I can bring server2 back online and try to investigate what is consuming that much memory, but my<br>
RabbitMQ/Erlang knowledge is not good enough, therefore reaching out for some help.<br>
<br>
So any help would be much appreciated.<br>
<br>
Thx<br>
Matthias<br>
<br>
Our setup is something like the following:<br>
2 servers exclusively running RabbitMQ on CentOS 5.x (high watermark ~22Gb),<br>
- both with web-console enabled<br>
- both defined as disk nodes<br>
- both running RabbitMQ 2.8.5 on Erlang R15B01 (after the upgrade, Erlang was already at R15 before)<br>
10 queues configured with mirroring<br>
3 queues configured (without mirroring) only on server1 with a TTL<br>
Most consumers are connecting to server1, server2 only in case of failover<br>
<br>
We get about 1k messages/sec (with peaks much higher than that) into the system, and each message is<br>
passed through several queues for processing.<br>
<br>
-bash-3.2$ sbin/rabbitmqctl eval 'erlang:memory().'<br>
[{total,<a href="tel:5445584424" value="+15445584424" target="_blank">5445584424</a>},<br>
{processes,<a href="tel:2184155418" value="+12184155418" target="_blank">2184155418</a>},<br>
{processes_used,<a href="tel:2184122352" value="+12184122352" target="_blank">2184122352</a>},<br>
{system,3261429006},<br>
{atom,703377},<br>
{atom_used,678425},<br>
{binary,<a href="tel:3216386480" value="+13216386480" target="_blank">3216386480</a>},<br>
{code,17978535},<br>
{ets,4142048}]<br>
...done.<br>
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
</blockquote></div><br>
</div></div></blockquote></div><br>