[rabbitmq-discuss] |Spam| Re: Possible memory leak in the management plugin
Travis Mehlinger
tmehlinger at gmail.com
Tue Jun 18 23:23:24 BST 2013
Hi Simon,
Sure thing, here you go:
https://gist.github.com/tmehlinger/158bb003fe3d64618a4e
Best, Travis
On Tue, Jun 18, 2013 at 10:47 AM, Simon MacMullen <simon at rabbitmq.com>wrote:
> OK, that definitely looks like a leak. Could you also give me the output
> from:
>
> rabbitmqctl eval '{[{T, ets:info(T,size), ets:info(T,memory)} || T <-
> lists:sort(ets:all()), rabbit_mgmt_db <- [ets:info(T, name)]],
> sys:get_status(global:whereis_**name(rabbit_mgmt_db))}.'
>
> to make sure I'm clear on which table is leaking.
>
> Cheers, Simon
>
> On 18/06/13 16:11, Travis Mehlinger wrote:
>
>> Hi Simon,
>>
>> We declare those queues as exclusive so they're getting cleaned up
>> automatically.
>>
>> I ran the command you gave periodically over the course of the last two
>> hours. The row count and total size in the highlighted line are
>> definitely growing unchecked. All other values hovered closely around
>> what you see in the gist.
>>
>> https://gist.github.com/**tmehlinger/**0c9a9a0d5fe1d31c8f6d#file-**
>> gistfile1-txt-L9<https://gist.github.com/tmehlinger/0c9a9a0d5fe1d31c8f6d#file-gistfile1-txt-L9>
>>
>> Thanks, Travis
>>
>>
>> On Tue, Jun 18, 2013 at 5:23 AM, Simon MacMullen <simon at rabbitmq.com
>> <mailto:simon at rabbitmq.com>> wrote:
>>
>> Hi. So I assume your monitoring code is not actually leaking those
>> queues - they are getting deleted I assume? How? (Are they
>> autodelete, exclusive, x-expires, deleted manually?)
>>
>> If so, can you run:
>>
>> rabbitmqctl eval '[{ets:info(T,size), ets:info(T,memory)} || T <-
>> lists:sort(ets:all()), rabbit_mgmt_db <- [ets:info(T, name)]].'
>>
>> periodically? This will output a list of tuples showing the rows and
>> bytes per table for each table in the mgmt DB. Do these increase?
>>
>> Cheers, Simon
>>
>>
>> On 17/06/13 20:08, Travis Mehlinger wrote:
>>
>> Hi Simon,
>>
>> I have more information for you. It turns out I hadn't fully
>> understood
>> the interaction causing this to happen.
>>
>> Aside from their regular communication, our services also declare
>> a
>> queue bound on # to an exchange that we use for collecting stats
>> the
>> services store internally. In addition to hitting the REST API for
>> information about the broker, the monitor also opens a
>> connection/channel, declares an anonymous queue for itself, then
>> sends a
>> message indicating to our services that they should respond with
>> their
>> statistics. The services then send a message with a routing key
>> that
>> will direct the response onto the queue declared by the monitor.
>> This
>> happens every five seconds.
>>
>> It appears that this is in fact responsible for memory consumption
>> growing out of control. If I disable that aspect of monitoring
>> and leave
>> the REST API monitor up, memory consumption stays level.
>>
>> The problem seems reminiscent of the issues described in this
>> mailing
>> list thread:
>> http://rabbitmq.1065348.n5.__n**abble.com/RabbitMQ-Queues-__**
>> memory-leak-td25813.html<http://nabble.com/RabbitMQ-Queues-__memory-leak-td25813.html>
>> <http://rabbitmq.1065348.n5.**nabble.com/RabbitMQ-Queues-**
>> memory-leak-td25813.html<http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-Queues-memory-leak-td25813.html>
>> >.
>> However, the queues we declare for stats collection are *not*
>> mirrored.
>>
>> Hope that helps narrow things down. :)
>>
>> Best, Travis
>>
>>
>> On Mon, Jun 17, 2013 at 12:58 PM, Travis Mehlinger
>> <tmehlinger at gmail.com <mailto:tmehlinger at gmail.com>
>> <mailto:tmehlinger at gmail.com <mailto:tmehlinger at gmail.com>>**>
>> wrote:
>>
>> Hi Simon,
>>
>> I flipped our monitor back on and let Rabbit consume some
>> additional
>> memory. Invoking the garbage collector had no impact.
>>
>> Let me know what further information you'd like to see and
>> I'll be
>> happy to provide it.
>>
>> Thanks, Travis
>>
>>
>> On Mon, Jun 17, 2013 at 10:32 AM, Simon MacMullen
>> <simon at rabbitmq.com <mailto:simon at rabbitmq.com>
>> <mailto:simon at rabbitmq.com <mailto:simon at rabbitmq.com>>> wrote:
>>
>> On 17/06/13 15:45, Travis Mehlinger wrote:
>>
>> Hi Simon,
>>
>> Thanks for getting back to me. I'll need to restart
>> our
>> monitor and give
>> it some time to leak the memory. I'll let you know
>> the
>> results sometime
>> later today.
>>
>> One thing I failed to mention in my initial report:
>> whenever we
>> restarted one of our services, the queues they were
>> using
>> would get
>> cleaned up (we have them set to auto-delete) and
>> redeclared.
>> Whenever we
>> did that, we would see the memory consumption of the
>> management DB fall
>> off sharply before starting to rise again.
>>
>>
>> That is presumably because the historical data the
>> management
>> plugin has been retaining for those queues got thrown
>> away. If
>> you don't want to retain this data at all, change the
>> configuration as documented here:
>>
>> http://www.rabbitmq.com/____**management.html#sample-____**
>> retention<http://www.rabbitmq.com/____management.html#sample-____retention>
>> <http://www.rabbitmq.com/__**management.html#sample-__**retention<http://www.rabbitmq.com/__management.html#sample-__retention>
>> >
>>
>>
>> <http://www.rabbitmq.com/__**management.html#sample-__**retention<http://www.rabbitmq.com/__management.html#sample-__retention>
>> <http://www.rabbitmq.com/**management.html#sample-**retention<http://www.rabbitmq.com/management.html#sample-retention>
>> >>
>>
>> However, I (currently) don't believe it's this
>> historical data
>> you are seeing as "leaking" since making queries should
>> not
>> cause any more of it to be retained.
>>
>> Cheers, Simon
>>
>> Let me know if you'd like any further information
>> in the
>> meantime.
>>
>> Best, Travis
>>
>>
>> On Mon, Jun 17, 2013 at 6:39 AM, Simon MacMullen
>> <simon at rabbitmq.com <mailto:simon at rabbitmq.com>
>> <mailto:simon at rabbitmq.com <mailto:simon at rabbitmq.com>>
>> <mailto:simon at rabbitmq.com
>> <mailto:simon at rabbitmq.com> <mailto:simon at rabbitmq.com
>> <mailto:simon at rabbitmq.com>>>> wrote:
>>
>> Hi. Thanks for the report.
>>
>> My first guess is that garbage collection for
>> the
>> management DB
>> process is happening too slowly. Can you invoke:
>>
>> $ rabbitmqctl eval
>>
>>
>> 'P=global:whereis_name(rabbit_**______mgmt_db),M1=process_**
>> info(__P,
>>
>> memory),garbage_collect(P),M2=**______process_info(P,
>> memory),{M1,M2,rabbit_vm:_____**_memory()}.'
>>
>>
>>
>> and post the results?
>>
>> Cheers, Simon
>>
>> On 15/06/13 03:09, Travis Mehlinger wrote:
>>
>> We recently upgraded RabbitMQ from 3.0.4
>> to 3.1.1
>> after noticing
>> two bug
>> fixes in 3.1.0 related to our RabbitMQ
>> deployment:
>>
>> * 25524 fix memory leak in mirror queue
>> slave
>> with many
>> short-lived
>> publishing channel
>> * 25290 fix per-queue memory leak
>> recording
>> stats for mirror
>> queue slaves
>>
>> However, in our case, it seems that the
>> management
>> plugin may
>> still have
>> a memory leak. We have a monitoring agent
>> that hits
>> the REST API to
>> gather information about the broker (number
>> of
>> queues, queue depth,
>> etc.). With the monitoring agent running
>> and making
>> requests
>> against the
>> API, memory consumption steadily
>> increased; when we
>> stopped the
>> agent,
>> memory consumption in the management
>> plugin leveled
>> off.
>>
>> Here a couple graphs detailing memory
>> consumption
>> in the broker (the
>> figures are parsed from rabbitmqctl
>> report). The
>> first graph
>> shows the
>> ebb and flow of memory consumption in a
>> number of
>> components and the
>> second shows just consumption by the
>> management
>> plugin. You can see
>> pretty clearly where we stopped the
>> monitoring
>> agent at 1:20.
>>
>> https://dl.dropboxusercontent.**______com/u/7022167/**
>> Screenshots/__n-____np6obt-**m9f.png
>>
>>
>> <https://dl.__dropboxuserconte**__nt.com/u/__7022167/__**
>> Screenshots/n-np6obt-__m9f.png<http://dropboxuserconte__nt.com/u/__7022167/__Screenshots/n-np6obt-__m9f.png>
>> <http://dropboxusercontent.**com/u/__7022167/Screenshots/n-**
>> np6obt-__m9f.png<http://dropboxusercontent.com/u/__7022167/Screenshots/n-np6obt-__m9f.png>
>> >
>>
>> <https://dl.__dropboxuserconte**nt.com/u/__7022167/**
>> Screenshots/n-np6obt-__m9f.png<http://dropboxusercontent.com/u/__7022167/Screenshots/n-np6obt-__m9f.png>
>> <https://dl.**dropboxusercontent.com/u/**
>> 7022167/Screenshots/n-np6obt-**m9f.png<https://dl.dropboxusercontent.com/u/7022167/Screenshots/n-np6obt-m9f.png>
>> >>>
>> https://dl.dropboxusercontent.**______com/u/7022167/**
>> Screenshots/______**an6dpup33xvx.png
>>
>>
>>
>> <https://dl.__dropboxuserconte**__nt.com/u/__7022167/__**
>> Screenshots/__an6dpup33xvx.png<http://dropboxuserconte__nt.com/u/__7022167/__Screenshots/__an6dpup33xvx.png>
>> <http://dropboxusercontent.**com/u/__7022167/Screenshots/__**
>> an6dpup33xvx.png<http://dropboxusercontent.com/u/__7022167/Screenshots/__an6dpup33xvx.png>
>> >
>>
>>
>> <https://dl.__dropboxuserconte**nt.com/u/__7022167/**
>> Screenshots/__an6dpup33xvx.png<http://dropboxusercontent.com/u/__7022167/Screenshots/__an6dpup33xvx.png>
>> <https://dl.**dropboxusercontent.com/u/**7022167/Screenshots/**
>> an6dpup33xvx.png<https://dl.dropboxusercontent.com/u/7022167/Screenshots/an6dpup33xvx.png>
>> >>>
>>
>> We have two clustered brokers, both running
>> RabbitMQ 3.1.1 on Erlang
>> R14B-04.1. There are typically around 200
>> queues,
>> about 20 of
>> which are
>> mirrored. There are generally about 200
>> consumers.
>> Messages are
>> rarely
>> queued and most queues typically sit idle.
>>
>> I'll be happy to provide any further
>> diagnostic
>> information.
>>
>> Thanks!
>>
>>
>>
>> ______________________________**_______________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.__rabbi**____tmq.com<http://rabbi____tmq.com>
>> <http://rabbi__tmq.com>
>> <http://rabbitmq.com>
>> <mailto:rabbitmq-discuss@
>> <mailto:rabbitmq-discuss@>__li**s__ts.rabbitmq.com<http://lis__ts.rabbitmq.com>
>> <http://lists.rabbitmq.com>
>> <mailto:rabbitmq-discuss at __lis**ts.rabbitmq.com<http://lists.rabbitmq.com>
>> <mailto:rabbitmq-discuss@**lists.rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
>> >>>
>> https://lists.rabbitmq.com/___**___cgi-bin/mailman/listinfo/__**
>> ____rabbitmq-discuss<https://lists.rabbitmq.com/______cgi-bin/mailman/listinfo/______rabbitmq-discuss>
>> <https://lists.rabbitmq.com/__**__cgi-bin/mailman/listinfo/___**
>> _rabbitmq-discuss<https://lists.rabbitmq.com/____cgi-bin/mailman/listinfo/____rabbitmq-discuss>
>> >
>>
>> <https://lists.rabbitmq.com/__**__cgi-bin/mailman/listinfo/___**
>> _rabbitmq-discuss<https://lists.rabbitmq.com/____cgi-bin/mailman/listinfo/____rabbitmq-discuss>
>> <https://lists.rabbitmq.com/__**cgi-bin/mailman/listinfo/__**
>> rabbitmq-discuss<https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss>
>> >>
>>
>>
>>
>>
>> <https://lists.rabbitmq.com/__**__cgi-bin/mailman/listinfo/___**
>> _rabbitmq-discuss<https://lists.rabbitmq.com/____cgi-bin/mailman/listinfo/____rabbitmq-discuss>
>> <https://lists.rabbitmq.com/__**cgi-bin/mailman/listinfo/__**
>> rabbitmq-discuss<https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss>
>> >
>>
>> <https://lists.rabbitmq.com/__**cgi-bin/mailman/listinfo/__**
>> rabbitmq-discuss<https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss>
>> <https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**
>> rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>> >>>
>>
>>
>>
>> --
>> Simon MacMullen
>> RabbitMQ, Pivotal
>>
>>
>>
>>
>> --
>> Simon MacMullen
>> RabbitMQ, Pivotal
>>
>>
>>
>>
>>
>> --
>> Simon MacMullen
>> RabbitMQ, Pivotal
>>
>>
>>
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130618/04ecfd56/attachment.htm>
More information about the rabbitmq-discuss
mailing list