[rabbitmq-discuss] RabbitMQ crashes hard when it runs out of memory

[Stephen Day]

>From your suggestions, it looks like I was on the right track. Output is
inline.

Moved to rabbitmq-discuss.

And again, your guidance is greatly appreciated.

-Stephen

On Thu, Oct 22, 2009 at 1:51 AM, Matthias Radestock <matthias at lshift.net>wrote:

> Stephen,
>
> Stephen Day wrote:
>
>> After running for few days, limiting queue size on producer side (no
>> memory backpressure), I have noticed that RabbitMQ has restarted several
>> times due to out of memory errors. Currently, I have about 150,000
>> persistent messages being consumed by 16 clients with qos. Over about 2-6
>> hours, the RSS and VM size of the erlang process grows unbounded, without
>> the producers adding any messages to the queue (externally limited to
>> 100,000), and the erlang process crashes. I am running version 1.7.0.
>>
>
> Memory consumption can increase due to gc effects, but not without bounds.
>
> How many messages does rabbit think there are in the queues - check with
> 'rabbitmqctl list_queues' - and how big are the messages?
>

$ ~/rabbitmq/sbin/rabbitmqctl list_queues name messages
messages_unacknowledged consumers memory
Listing queues ...
<redacted>    0    0    0    2376
<redacted>    27    0    0    27572
<redacted>    0    0    1    2992
<redacted>    149917    16    16    71106232

The messages are no bigger than 512 bytes, but vary in size.


>
> How large is the persister log?


$ du -b mnesia/rabbit/*
162    mnesia/rabbit/DECISION_TAB.LOG
98    mnesia/rabbit/LATEST.LOG
8    mnesia/rabbit/rabbit_config.DCD
995    mnesia/rabbit/rabbit_durable_exchange.DCD
606    mnesia/rabbit/rabbit_durable_queue.DCD
976    mnesia/rabbit/rabbit_durable_queue.DCL
1396    mnesia/rabbit/rabbit_durable_route.DCD
16720    mnesia/rabbit/rabbit_durable_route.DCL
82515919    mnesia/rabbit/rabbit_persister.LOG
82526910    mnesia/rabbit/rabbit_persister.LOG.previous
4    mnesia/rabbit/rabbit_serial
134    mnesia/rabbit/rabbit_user.DCD
192    mnesia/rabbit/rabbit_user_permission.DCD
133    mnesia/rabbit/rabbit_vhost.DCD
12383    mnesia/rabbit/schema.DAT


>
>  Now, I was under the impression that this memory problem was due to the
>> number of messages being added, but this may not be the case. What
>> information do you need to troubleshoot this? Are there any erlang tools,
>> similar to valgrind, that can track down this memory usage?
>>
>
> There are plenty of tools to figure out what's going on in a running Erlang
> system. You need to know a fair bit about Erlang to use them though.
>
> As a starting point, remsh into rabbit with "erl -sname sh -remsh rabbit@<host>"
> (you'll need to run this as the 'rabbitmq' user in a typical rabbit
> installation) and on the prompt type
>  memory().
> which will tell you at a coarse-grain level where the memory has gone.
>

Here is the memory() output:

(rabbit at vs-dfw-ctl11)1> memory().
[{total,1015558480},
 {processes,161498128},
 {processes_used,161494240},
 {system,854060352},
 {atom,516217},
 {atom_used,492146},
 {binary,778446312},
 {code,3860276},
 {ets,70066180}]

Right now, the system is using around 1GB of memory, mostly concentrated in
binary allocation. I was looking at this yesterday as well, and the process
memory hasn't changed much.


>
>
> Regards,
>
> Matthias.
> PS: could we move this discussion onto rabbitmq-discuss, please?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20091022/eed2a2e9/attachment.htm