<div dir="ltr">Hi Simon,<div><br></div><div>Sure thing, here you go: <a href="https://gist.github.com/tmehlinger/158bb003fe3d64618a4e">https://gist.github.com/tmehlinger/158bb003fe3d64618a4e</a></div><div><br></div><div style>
Best, Travis</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jun 18, 2013 at 10:47 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">OK, that definitely looks like a leak. Could you also give me the output from:<br>
<br>
rabbitmqctl eval '{[{T, ets:info(T,size), ets:info(T,memory)} || T <- lists:sort(ets:all()), rabbit_mgmt_db <- [ets:info(T, name)]], sys:get_status(global:whereis_<u></u>name(rabbit_mgmt_db))}.'<br>
<br>
to make sure I'm clear on which table is leaking.<br>
<br>
Cheers, Simon<br>
<br>
On 18/06/13 16:11, Travis Mehlinger wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Simon,<br>
<br>
We declare those queues as exclusive so they're getting cleaned up<br>
automatically.<br>
<br>
I ran the command you gave periodically over the course of the last two<br>
hours. The row count and total size in the highlighted line are<br>
definitely growing unchecked. All other values hovered closely around<br>
what you see in the gist.<br>
<br>
<a href="https://gist.github.com/tmehlinger/0c9a9a0d5fe1d31c8f6d#file-gistfile1-txt-L9" target="_blank">https://gist.github.com/<u></u>tmehlinger/<u></u>0c9a9a0d5fe1d31c8f6d#file-<u></u>gistfile1-txt-L9</a><br>
<br>
Thanks, Travis<br>
<br>
<br>
On Tue, Jun 18, 2013 at 5:23 AM, Simon MacMullen <<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a><br>
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>>> wrote:<br>
<br>
Hi. So I assume your monitoring code is not actually leaking those<br>
queues - they are getting deleted I assume? How? (Are they<br>
autodelete, exclusive, x-expires, deleted manually?)<br>
<br>
If so, can you run:<br>
<br>
rabbitmqctl eval '[{ets:info(T,size), ets:info(T,memory)} || T <-<br>
lists:sort(ets:all()), rabbit_mgmt_db <- [ets:info(T, name)]].'<br>
<br>
periodically? This will output a list of tuples showing the rows and<br>
bytes per table for each table in the mgmt DB. Do these increase?<br>
<br>
Cheers, Simon<br>
<br>
<br>
On 17/06/13 20:08, Travis Mehlinger wrote:<br>
<br>
Hi Simon,<br>
<br>
I have more information for you. It turns out I hadn't fully<br>
understood<br>
the interaction causing this to happen.<br>
<br>
Aside from their regular communication, our services also declare a<br>
queue bound on # to an exchange that we use for collecting stats the<br>
services store internally. In addition to hitting the REST API for<br>
information about the broker, the monitor also opens a<br>
connection/channel, declares an anonymous queue for itself, then<br>
sends a<br>
message indicating to our services that they should respond with<br>
their<br>
statistics. The services then send a message with a routing key that<br>
will direct the response onto the queue declared by the monitor.<br>
This<br>
happens every five seconds.<br>
<br>
It appears that this is in fact responsible for memory consumption<br>
growing out of control. If I disable that aspect of monitoring<br>
and leave<br>
the REST API monitor up, memory consumption stays level.<br>
<br>
The problem seems reminiscent of the issues described in this<br>
mailing<br>
list thread:<br>
<a href="http://rabbitmq.1065348.n5." target="_blank">http://rabbitmq.1065348.n5.</a>__<a href="http://nabble.com/RabbitMQ-Queues-__memory-leak-td25813.html" target="_blank">n<u></u>abble.com/RabbitMQ-Queues-__<u></u>memory-leak-td25813.html</a><br>
<<a href="http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-Queues-memory-leak-td25813.html" target="_blank">http://rabbitmq.1065348.n5.<u></u>nabble.com/RabbitMQ-Queues-<u></u>memory-leak-td25813.html</a>>.<br>
However, the queues we declare for stats collection are *not*<br>
mirrored.<br>
<br>
Hope that helps narrow things down. :)<br>
<br>
Best, Travis<br>
<br>
<br>
On Mon, Jun 17, 2013 at 12:58 PM, Travis Mehlinger<br>
<<a href="mailto:tmehlinger@gmail.com" target="_blank">tmehlinger@gmail.com</a> <mailto:<a href="mailto:tmehlinger@gmail.com" target="_blank">tmehlinger@gmail.com</a>><br>
<mailto:<a href="mailto:tmehlinger@gmail.com" target="_blank">tmehlinger@gmail.com</a> <mailto:<a href="mailto:tmehlinger@gmail.com" target="_blank">tmehlinger@gmail.com</a>>><u></u>> wrote:<br>
<br>
Hi Simon,<br>
<br>
I flipped our monitor back on and let Rabbit consume some<br>
additional<br>
memory. Invoking the garbage collector had no impact.<br>
<br>
Let me know what further information you'd like to see and<br>
I'll be<br>
happy to provide it.<br>
<br>
Thanks, Travis<br>
<br>
<br>
On Mon, Jun 17, 2013 at 10:32 AM, Simon MacMullen<br>
<<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a> <mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>><br>
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a> <mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>>>> wrote:<br>
<br>
On 17/06/13 15:45, Travis Mehlinger wrote:<br>
<br>
Hi Simon,<br>
<br>
Thanks for getting back to me. I'll need to restart our<br>
monitor and give<br>
it some time to leak the memory. I'll let you know the<br>
results sometime<br>
later today.<br>
<br>
One thing I failed to mention in my initial report:<br>
whenever we<br>
restarted one of our services, the queues they were<br>
using<br>
would get<br>
cleaned up (we have them set to auto-delete) and<br>
redeclared.<br>
Whenever we<br>
did that, we would see the memory consumption of the<br>
management DB fall<br>
off sharply before starting to rise again.<br>
<br>
<br>
That is presumably because the historical data the<br>
management<br>
plugin has been retaining for those queues got thrown<br>
away. If<br>
you don't want to retain this data at all, change the<br>
configuration as documented here:<br>
<br>
<a href="http://www.rabbitmq.com/____management.html#sample-____retention" target="_blank">http://www.rabbitmq.com/____<u></u>management.html#sample-____<u></u>retention</a><br>
<<a href="http://www.rabbitmq.com/__management.html#sample-__retention" target="_blank">http://www.rabbitmq.com/__<u></u>management.html#sample-__<u></u>retention</a>><br>
<br>
<br>
<<a href="http://www.rabbitmq.com/__management.html#sample-__retention" target="_blank">http://www.rabbitmq.com/__<u></u>management.html#sample-__<u></u>retention</a><br>
<<a href="http://www.rabbitmq.com/management.html#sample-retention" target="_blank">http://www.rabbitmq.com/<u></u>management.html#sample-<u></u>retention</a>>><br>
<br>
However, I (currently) don't believe it's this<br>
historical data<br>
you are seeing as "leaking" since making queries should not<br>
cause any more of it to be retained.<br>
<br>
Cheers, Simon<br>
<br>
Let me know if you'd like any further information<br>
in the<br>
meantime.<br>
<br>
Best, Travis<br>
<br>
<br>
On Mon, Jun 17, 2013 at 6:39 AM, Simon MacMullen<br>
<<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a> <mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>><br>
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a> <mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>>><br>
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a><br>
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>> <mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a><br>
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>>>>> wrote:<br>
<br>
Hi. Thanks for the report.<br>
<br>
My first guess is that garbage collection for the<br>
management DB<br>
process is happening too slowly. Can you invoke:<br>
<br>
$ rabbitmqctl eval<br>
<br>
<br>
'P=global:whereis_name(rabbit_<u></u>______mgmt_db),M1=process_<u></u>info(__P,<br>
<br>
memory),garbage_collect(P),M2=<u></u>______process_info(P,<br>
memory),{M1,M2,rabbit_vm:_____<u></u>_memory()}.'<br>
<br>
<br>
<br>
and post the results?<br>
<br>
Cheers, Simon<br>
<br>
On 15/06/13 03:09, Travis Mehlinger wrote:<br>
<br>
We recently upgraded RabbitMQ from 3.0.4<br>
to 3.1.1<br>
after noticing<br>
two bug<br>
fixes in 3.1.0 related to our RabbitMQ<br>
deployment:<br>
<br>
* 25524 fix memory leak in mirror queue<br>
slave<br>
with many<br>
short-lived<br>
publishing channel<br>
* 25290 fix per-queue memory leak recording<br>
stats for mirror<br>
queue slaves<br>
<br>
However, in our case, it seems that the<br>
management<br>
plugin may<br>
still have<br>
a memory leak. We have a monitoring agent<br>
that hits<br>
the REST API to<br>
gather information about the broker (number of<br>
queues, queue depth,<br>
etc.). With the monitoring agent running<br>
and making<br>
requests<br>
against the<br>
API, memory consumption steadily<br>
increased; when we<br>
stopped the<br>
agent,<br>
memory consumption in the management<br>
plugin leveled<br>
off.<br>
<br>
Here a couple graphs detailing memory<br>
consumption<br>
in the broker (the<br>
figures are parsed from rabbitmqctl<br>
report). The<br>
first graph<br>
shows the<br>
ebb and flow of memory consumption in a<br>
number of<br>
components and the<br>
second shows just consumption by the<br>
management<br>
plugin. You can see<br>
pretty clearly where we stopped the monitoring<br>
agent at 1:20.<br>
<br>
<a href="https://dl.dropboxusercontent." target="_blank">https://dl.dropboxusercontent.</a><u></u>______com/u/7022167/<u></u>Screenshots/__n-____np6obt-<u></u>m9f.png<br>
<br>
<br>
<<a href="https://dl." target="_blank">https://dl.</a>__<a href="http://dropboxuserconte__nt.com/u/__7022167/__Screenshots/n-np6obt-__m9f.png" target="_blank">dropboxuserconte<u></u>__nt.com/u/__7022167/__<u></u>Screenshots/n-np6obt-__m9f.png</a><br>
<<a href="http://dropboxusercontent.com/u/__7022167/Screenshots/n-np6obt-__m9f.png" target="_blank">http://dropboxusercontent.<u></u>com/u/__7022167/Screenshots/n-<u></u>np6obt-__m9f.png</a>><br>
<br>
<<a href="https://dl." target="_blank">https://dl.</a>__<a href="http://dropboxusercontent.com/u/__7022167/Screenshots/n-np6obt-__m9f.png" target="_blank">dropboxuserconte<u></u>nt.com/u/__7022167/<u></u>Screenshots/n-np6obt-__m9f.png</a><br>
<<a href="https://dl.dropboxusercontent.com/u/7022167/Screenshots/n-np6obt-m9f.png" target="_blank">https://dl.<u></u>dropboxusercontent.com/u/<u></u>7022167/Screenshots/n-np6obt-<u></u>m9f.png</a>>>><br>
<a href="https://dl.dropboxusercontent." target="_blank">https://dl.dropboxusercontent.</a><u></u>______com/u/7022167/<u></u>Screenshots/______<u></u>an6dpup33xvx.png<br>
<br>
<br>
<br>
<<a href="https://dl." target="_blank">https://dl.</a>__<a href="http://dropboxuserconte__nt.com/u/__7022167/__Screenshots/__an6dpup33xvx.png" target="_blank">dropboxuserconte<u></u>__nt.com/u/__7022167/__<u></u>Screenshots/__an6dpup33xvx.png</a><br>
<<a href="http://dropboxusercontent.com/u/__7022167/Screenshots/__an6dpup33xvx.png" target="_blank">http://dropboxusercontent.<u></u>com/u/__7022167/Screenshots/__<u></u>an6dpup33xvx.png</a>><br>
<br>
<br>
<<a href="https://dl." target="_blank">https://dl.</a>__<a href="http://dropboxusercontent.com/u/__7022167/Screenshots/__an6dpup33xvx.png" target="_blank">dropboxuserconte<u></u>nt.com/u/__7022167/<u></u>Screenshots/__an6dpup33xvx.png</a><br>
<<a href="https://dl.dropboxusercontent.com/u/7022167/Screenshots/an6dpup33xvx.png" target="_blank">https://dl.<u></u>dropboxusercontent.com/u/<u></u>7022167/Screenshots/<u></u>an6dpup33xvx.png</a>>>><br>
<br>
We have two clustered brokers, both running<br>
RabbitMQ 3.1.1 on Erlang<br>
R14B-04.1. There are typically around 200<br>
queues,<br>
about 20 of<br>
which are<br>
mirrored. There are generally about 200<br>
consumers.<br>
Messages are<br>
rarely<br>
queued and most queues typically sit idle.<br>
<br>
I'll be happy to provide any further<br>
diagnostic<br>
information.<br>
<br>
Thanks!<br>
<br>
<br>
<br>
______________________________<u></u>_______________________<br>
rabbitmq-discuss mailing list<br>
rabbitmq-discuss@lists.__<a href="http://rabbi____tmq.com" target="_blank">rabbi<u></u>____tmq.com</a><br>
<<a href="http://rabbi__tmq.com" target="_blank">http://rabbi__tmq.com</a>><br>
<<a href="http://rabbitmq.com" target="_blank">http://rabbitmq.com</a>><br>
<mailto:<a href="mailto:rabbitmq-discuss@" target="_blank">rabbitmq-discuss@</a><br>
<mailto:<a href="mailto:rabbitmq-discuss@" target="_blank">rabbitmq-discuss@</a>>__<a href="http://lis__ts.rabbitmq.com" target="_blank">li<u></u>s__ts.rabbitmq.com</a><br>
<<a href="http://lists.rabbitmq.com" target="_blank">http://lists.rabbitmq.com</a>><br>
<mailto:<a href="mailto:rabbitmq-discuss@" target="_blank">rabbitmq-discuss@</a>__<a href="http://lists.rabbitmq.com" target="_blank">lis<u></u>ts.rabbitmq.com</a><br>
<mailto:<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@<u></u>lists.rabbitmq.com</a>>>><br>
<a href="https://lists.rabbitmq.com/______cgi-bin/mailman/listinfo/______rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/___<u></u>___cgi-bin/mailman/listinfo/__<u></u>____rabbitmq-discuss</a><br>
<<a href="https://lists.rabbitmq.com/____cgi-bin/mailman/listinfo/____rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/__<u></u>__cgi-bin/mailman/listinfo/___<u></u>_rabbitmq-discuss</a>><br>
<br>
<<a href="https://lists.rabbitmq.com/____cgi-bin/mailman/listinfo/____rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/__<u></u>__cgi-bin/mailman/listinfo/___<u></u>_rabbitmq-discuss</a><br>
<<a href="https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/__<u></u>cgi-bin/mailman/listinfo/__<u></u>rabbitmq-discuss</a>>><br>
<br>
<br>
<br>
<br>
<<a href="https://lists.rabbitmq.com/____cgi-bin/mailman/listinfo/____rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/__<u></u>__cgi-bin/mailman/listinfo/___<u></u>_rabbitmq-discuss</a><br>
<<a href="https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/__<u></u>cgi-bin/mailman/listinfo/__<u></u>rabbitmq-discuss</a>><br>
<br>
<<a href="https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/__<u></u>cgi-bin/mailman/listinfo/__<u></u>rabbitmq-discuss</a><br>
<<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a>>>><br>
<br>
<br>
<br>
--<br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
<br>
<br>
<br>
<br>
--<br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
<br>
<br>
<br>
<br>
<br>
--<br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
<br>
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
</font></span></blockquote></div><br></div>