[rabbitmq-discuss] rabbitmq-server crashes hard while consuming 31GB of RAM

Wed Nov 23 16:40:54 GMT 2011

On 23/11/11 15:05, Muharem Hrnjadovic wrote:
> Please see the "sudo rabbitmqctl report" link on:
>      https://bugs.launchpad.net/openquake/+bug/894024/comments/1

Hi. Thanks for this.

There are a number of interesting things going on:

1) Your "rabbitmqctl report" invocation shows about 28000 channels open 
and no connections. Did you edit connection details out of this?

2) About 9GB of memory is used by binaries. This may indicate that you 
have some large messages around - are your messages particularly large?

3) While the memory alarm was set and cleared a number of times, in the 
end the Erlang VM crashed with:

   no next heap size found: -2106532246, offset 0
   Aborted

This error message indicates that the Erlang VM was not available to 
increase the heap size of a particular process as it ran off the end of 
a table of permitted heap sizes. On a 64GB machine the maximum heap size 
is 6.6 Exabytes: I hope you did not run in to that. Unfortunately there 
is a bug in Erlang R13 which garbles the error message above (hence why 
the alleged heap size is a negative number). So:

3a) is this a 64 bit machine?

3b) Can you run:

su rabbitmq -s /bin/sh -c 'erl -remsh rabbit@`hostname -s` -sname foo 
-eval "io:format(\"~p\", 
[lists:sublist(lists:reverse(lists:sort([{process_info(Pid, memory), 
Pid, process_info(Pid)} || Pid <- processes()])), 30)]), halt()."'

(all on one line, as root) when rabbit is eating a lot of memory and 
send us the output? This should let us see if there are individual 
processes that are eating huge amounts of memory.

3c) Can you upgrade the Erlang VM to see if we can get a better version 
of that error message? Updated debs are available at:

  http://www.erlang-solutions.com/section/132/erlang-otp-packages

Due to limitations in dpkg you might need to:

# apt-get remove erlang-base
# dpkg -i esl-erlang_14.b.4-1_amd64.deb
# apt-get install erlang-nox
# dpkg -i rabbitmq-server_2.7.0-1_all.deb

(Yes, this is less than ideal but we've only just found out about the 
ESL packages...)

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware