[rabbitmq-discuss] RabbitMQ crashes hard when it runs out of memory

[Matthias Radestock]

Stephen,

Stephen Day wrote:
> $ ~/rabbitmq/sbin/rabbitmqctl list_queues name messages
> [...]
> <redacted>    149917    16    16    71106232
> du -b mnesia/rabbit/*
> [...]
> 82515919    mnesia/rabbit/rabbit_persister.LOG
> 82526910    mnesia/rabbit/rabbit_persister.LOG.previous

That all looks reasonable.

> Here is the memory() output:
> 
> (rabbit at vs-dfw-ctl11)1> memory().
> [{total,1015558480},
>  {processes,161498128},
>  {processes_used,161494240},
>  {system,854060352},
>  {atom,516217},
>  {atom_used,492146},
>  {binary,778446312},
>  {code,3860276},
>  {ets,70066180}]

Interesting. There is far more binary data allocated than you have
message data in your system.

We've noticed in the past that binaries can take quite a while to get
collected by the gc. One experiment you may want to try is to run
  garbage_collect(whereis(rabbit_persister)).
from the Erlang shell and see what difference that makes to the memory
report.

Also, the output of
   process_info(whereis(rabbit_persister)).
might be useful.

> Right now, the system is using around 1GB of memory

And you are saying that figure keeps growing, and eventually RabbitMQ
runs out of memory, even though the figures reported by 'rabbitmqctl
list_queues' and 'du' remain roughly constant? That's strange indeed.


Regards,

Matthias.