[rabbitmq-discuss] Memory leak with file_io_server, server_loop?

[Travis]

On Mon, Nov 5, 2012 at 6:23 PM, Matthias Radestock
<matthias at rabbitmq.com> wrote:
> Travis,
>
>
> On 05/11/12 23:38, Travis wrote:
>
>
> Hmm. forcing a gc of the file_io_server process *should* drop the memory
> usage. What exactly did you try?

I tried various forms of the following which didn't work (because of
ERL syntax issues)

sudo rabbitmqctl eval 'erlang:garbage_collect(whereis(file_io_server)).'
sudo rabbitmqctl eval 'erlang:garbage_collect(whereis(server_loop)).'

and then attempted

sudo rabbitmqctl eval 'erlang:garbage_collect().'

which returned true, but doesn't appear to have done anything.

I'm basically poking around in the dark here with ERL so I'm not
exactly sure if that does what I intend it to do (force an overall GC
to occur ... guessing no).

>
> rabbitmqctl eval '[garbage_collect(P) || {P, _} <-
> ets:tab2list(file_io_servers)].'
>
> should do the trick.
>

That did it.  Usage dropped by 400MB.  Then I found that error_logger
was chewing up ~200MB, so I ran the garbage_collect version y'all gave
me previously to handle that:

rabbitmqctl eval 'erlang:garbage_collect(whereis(error_logger)).'

and things dropped further.

> It is quite possible that the file_io_servers in question deal with log
> files. Do you have particularly large log files on those machines? You may
> want to reduce the connection log level - see
> http://www.rabbitmq.com/configure.html#config-items - and/or rotate the logs
> with http://www.rabbitmq.com/man/rabbitmqctl.1.man.html#rotate_logs
>

I wouldn't call them appreciably large.  We're using the standard
logrotate.d config that comes with the RPMs which rotates weekly.

On one server, last week's log file was 54MB, where as this week's
(with ~3 days of logs) is 19MB.

The log from this week breaks down with:

[tcampbell at host rabbitmq]$ grep REPORT rabbit\@host.log | awk -F====
'{print $1}' | sort | uniq -c
    116 =ERROR REPORT
  93656 =INFO REPORT
  40792 =WARNING REPORT

of which, all of the warnings are just errors about connections
closing abruptly.

closing AMQP connection <0.2945.6> (127.0.0.1:57242 -> 127.0.0.1:5672):
connection_closed_abruptly

and the info reports are the logs for accepting or closing connections
from localhost.

Travis


-- 
Travis Campbell
travis at ghostar.org