[rabbitmq-discuss] mysterious rabbit crash

[Dmitriy Samovskiy]

Hi all,

One of my rabbits running on an old machine crashed last night.

Starting yesterday at 3:04pm, rabbit apparently has been complaining about "alarm_handler: 
{set,{system_memory_high_watermark,[]}}" in the log file.

But at 00:27am this morning, it rolled the persister log and crashed.

Please note that the size of rabbit_persister.LOG.previous is roughly the same as the 
amount of memory it failed to allocate.

My question is is this a coincidence that the numbers are close? I would think rabbit has 
no use for *previous* persister log in this situation.

Disk where mnesia and persister log are located was not full at the time of crash (I 
think). I am also nearly positive that consumers were around just before and during the 
crash, and were properly consuming from the broker (I have charts with data going all the 
way to roughly 00:27am). Which makes me wonder why rabbit was using so much memory if its 
queues were being drained normally...

-rw-r--r--    1 rabbitmq rabbitmq 22722770 May 19 00:27 rabbit_persister.LOG.previous

=erl_crash_dump:0.1
Tue May 19 00:27:27 2009
Slogan: binary_alloc: Cannot allocate 22615166 bytes of memory (of type "binary"
).
System version: Erlang (BEAM) emulator version 5.6.5 [source] [async-threads:30]
Compiled: Fri Apr  3 11:49:36 2009
Atoms: 9807
=memory
total: 931495488
processes: 90885644
processes_used: 90882996
system: 840609844
atom: 435481
atom_used: 426079
binary: 815315560
code: 3451735
ets: 20791156


{rabbit,"RabbitMQ","1.5.4"},
                         {mnesia,"MNESIA  CXC 138 12","4.4.7"},
                         {os_mon,"CPO  CXC 138 46","2.1.8"},
                         {sasl,"SASL  CXC 138 11","2.1.5.4"},
                         {stdlib,"ERTS  CXC 138 10","1.15.5"},
                         {kernel,"ERTS  CXC 138 10","2.12.5"}]},

Has anybody seen anything like this?

This is a very low priority issue for me, I am just curious.

Thanks in advance for your thoughts and time spent looking into it.

- Dmitriy