[rabbitmq-discuss] RabbitMQ crashes hard when it runs out of memory
sjaday at gmail.com
Thu Oct 22 19:03:56 BST 2009
>From your suggestions, it looks like I was on the right track. Output is
Moved to rabbitmq-discuss.
And again, your guidance is greatly appreciated.
On Thu, Oct 22, 2009 at 1:51 AM, Matthias Radestock <matthias at lshift.net>wrote:
> Stephen Day wrote:
>> After running for few days, limiting queue size on producer side (no
>> memory backpressure), I have noticed that RabbitMQ has restarted several
>> times due to out of memory errors. Currently, I have about 150,000
>> persistent messages being consumed by 16 clients with qos. Over about 2-6
>> hours, the RSS and VM size of the erlang process grows unbounded, without
>> the producers adding any messages to the queue (externally limited to
>> 100,000), and the erlang process crashes. I am running version 1.7.0.
> Memory consumption can increase due to gc effects, but not without bounds.
> How many messages does rabbit think there are in the queues - check with
> 'rabbitmqctl list_queues' - and how big are the messages?
$ ~/rabbitmq/sbin/rabbitmqctl list_queues name messages
messages_unacknowledged consumers memory
Listing queues ...
<redacted> 0 0 0 2376
<redacted> 27 0 0 27572
<redacted> 0 0 1 2992
<redacted> 149917 16 16 71106232
The messages are no bigger than 512 bytes, but vary in size.
> How large is the persister log?
$ du -b mnesia/rabbit/*
> Now, I was under the impression that this memory problem was due to the
>> number of messages being added, but this may not be the case. What
>> information do you need to troubleshoot this? Are there any erlang tools,
>> similar to valgrind, that can track down this memory usage?
> There are plenty of tools to figure out what's going on in a running Erlang
> system. You need to know a fair bit about Erlang to use them though.
> As a starting point, remsh into rabbit with "erl -sname sh -remsh rabbit@<host>"
> (you'll need to run this as the 'rabbitmq' user in a typical rabbit
> installation) and on the prompt type
> which will tell you at a coarse-grain level where the memory has gone.
Here is the memory() output:
(rabbit at vs-dfw-ctl11)1> memory().
Right now, the system is using around 1GB of memory, mostly concentrated in
binary allocation. I was looking at this yesterday as well, and the process
memory hasn't changed much.
> PS: could we move this discussion onto rabbitmq-discuss, please?
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss