<div dir="ltr">Hi Simon,<div><br></div><div style>Thanks for getting back to me. I'll need to restart our monitor and give it some time to leak the memory. I'll let you know the results sometime later today.</div><div style>
<br></div><div style>One thing I failed to mention in my initial report: whenever we restarted one of our services, the queues they were using would get cleaned up (we have them set to auto-delete) and redeclared. Whenever we did that, we would see the memory consumption of the management DB fall off sharply before starting to rise again.</div>
<div style><br></div><div style>Let me know if you'd like any further information in the meantime.</div><div style><br></div><div style>Best, Travis</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Mon, Jun 17, 2013 at 6:39 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi. Thanks for the report.<br>
<br>
My first guess is that garbage collection for the management DB process is happening too slowly. Can you invoke:<br>
<br>
$ rabbitmqctl eval 'P=global:whereis_name(rabbit_<u></u>mgmt_db),M1=process_info(P, memory),garbage_collect(P),M2=<u></u>process_info(P, memory),{M1,M2,rabbit_vm:<u></u>memory()}.'<br>
<br>
and post the results?<br>
<br>
Cheers, Simon<br>
<br>
On 15/06/13 03:09, Travis Mehlinger wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
We recently upgraded RabbitMQ from 3.0.4 to 3.1.1 after noticing two bug<br>
fixes in 3.1.0 related to our RabbitMQ deployment:<br>
<br>
* 25524 fix memory leak in mirror queue slave with many short-lived<br>
publishing channel<br>
* 25290 fix per-queue memory leak recording stats for mirror queue slaves<br>
<br>
However, in our case, it seems that the management plugin may still have<br>
a memory leak. We have a monitoring agent that hits the REST API to<br>
gather information about the broker (number of queues, queue depth,<br>
etc.). With the monitoring agent running and making requests against the<br>
API, memory consumption steadily increased; when we stopped the agent,<br>
memory consumption in the management plugin leveled off.<br>
<br>
Here a couple graphs detailing memory consumption in the broker (the<br>
figures are parsed from rabbitmqctl report). The first graph shows the<br>
ebb and flow of memory consumption in a number of components and the<br>
second shows just consumption by the management plugin. You can see<br>
pretty clearly where we stopped the monitoring agent at 1:20.<br>
<br>
<a href="https://dl.dropboxusercontent.com/u/7022167/Screenshots/n-np6obt-m9f.png" target="_blank">https://dl.dropboxusercontent.<u></u>com/u/7022167/Screenshots/n-<u></u>np6obt-m9f.png</a><br>
<a href="https://dl.dropboxusercontent.com/u/7022167/Screenshots/an6dpup33xvx.png" target="_blank">https://dl.dropboxusercontent.<u></u>com/u/7022167/Screenshots/<u></u>an6dpup33xvx.png</a><br>
<br>
We have two clustered brokers, both running RabbitMQ 3.1.1 on Erlang<br>
R14B-04.1. There are typically around 200 queues, about 20 of which are<br>
mirrored. There are generally about 200 consumers. Messages are rarely<br>
queued and most queues typically sit idle.<br>
<br>
I'll be happy to provide any further diagnostic information.<br>
<br>
Thanks!<br>
<br>
<br>
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
</font></span></blockquote></div><br></div>