<div class="gmail_quote">On Thu, Sep 24, 2009 at 4:57 AM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@lshift.net">matthias@lshift.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Gavin,<br>
<div class="im"><br>
</div>Next time this happens I suggest you gather some diagnostics with<br>
rabbitmqctl list_queues name messages_ready messages_unacknowledged<br>
consumers<br></blockquote><div><br></div><div>I made sure there were no messages in the stack, but since it's still hanging out with the memory allocated:</div><div> </div><div><div>[root@mq07 ~]# /opt/rabbitmq/sbin/rabbitmqctl list_queues name messages_ready messages_unacknowledged</div>
<div>Listing queues ...</div><div>Notification<span class="Apple-tab-span" style="white-space:pre"> </span>0<span class="Apple-tab-span" style="white-space:pre"> </span>0</div><div>...done.</div><div><br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
If the consumer count is 0 then there are no consumers. If you have a<br>
consumer count above 0 and a messages_ready count above 0, and the<br>
broker appears to be idle, then that is a good indication that the<br>
consumers are there but stuck and the tcp connection to them is backlogged.<br></blockquote><div><br></div><div>Yeah I pulled off the consumers when it was done to see if that was why the memory wasn't given back.</div>
<div><br></div><div>lsof and netstat do not indicate any open sockets (in any state) to Rabbit from the consumer box.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">It is quite common for a gc'ed VM to hold on to large chunks of memory</div>
instead of handing them back to the OS.<br></blockquote><div><br></div><div>Is it safe to assume the VM will make use of what it's holding on to and it's not a leak of some sort?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Are you monitoring queues with rabbitmqctl or Alice? If that happens<br>
more frequently than once per second then the queue processes are kept<br>
active, which delays the freeing up of memory.<br></blockquote><div><br></div><div>Primarily Alice, and the monitor interval I am using is ~10 seconds.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
To get some diagnostics on where the memory has gone, shell into rabbit<br>
(erl -sname shell -remsh rabbit@<hostname>) and run<br>
memory().<br></blockquote><div><br></div><div>[root@mq07 ~]# erl -sname shell -remsh rabbit@mq07</div><div>Erlang R13B (erts-5.7.1) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:0] [hipe] [kernel-poll:false]</div><div>
<br></div><div>Eshell V5.7.1 (abort with ^G)</div><div>(rabbit@mq07)1> memory().</div><div>[{total,354514080},</div><div> {processes,23750072},</div><div> {processes_used,22779304},</div><div> {system,330764008},</div>
<div> {atom,775705},</div><div> {atom_used,748674},</div><div> {binary,22063432},</div><div> {code,7638629},</div><div> {ets,5241416}]</div><div>(rabbit@mq07)2> </div><div><br></div><div>Thanks,</div><div><br></div><div>
Gavin</div></div>