[rabbitmq-discuss] Erlang crashes trying to allocate 583848200 bytes of memory

[Marek Majkowski]

On Mon, Jan 17, 2011 at 10:20, Mark Hudson <markrogerhudson at gmail.com> wrote:
> Hi, I'm experiencing reproducible crashes of Erlang when running
> RabbitMQ 2.20 on a clean install of 64 bit Windows server 2008 R2 box
>
> In particular I get
> "broker running
> Crash dump was written to: erl_crash.dump
> eheap_alloc: Cannot allocate 583848200 bytes of memory (of type "old_heap").
> This application has requested the Runtime to terminate it in an unusual way.
> Please contact the application's support team for more information."
> "Cannot allocate 583848200 bytes of memory (of type "old_heap")." is
> the same message talked about in the previous discussion.
> I find that rabbitmqctl.bat takes a while to
> return there are no queues, then the erl.exe process grows to about
> 650 meg before erland crashes dumping out

The problem is that although you're using 64 bit windows,
there is no 64 bit Erlang for windows.
Erlang in windows runs as 32-bit program.

Looking here:
http://msdn.microsoft.com/en-us/library/aa366778(v=VS.85).aspx
it looks like by default, without IMAGE_FILE_LARGE_ADDRESS_AWARE flag set,
32 bit processes can use only 2 GB of RAM.

Additionally, Erlang garbage collector works like that:
 - allocate a large chunk of continuous memory
 - rewrite data (skipping unused garbage)
 - free old memory

As a result, to run Erlang you need to always stay
below 50% available memory (to have other 50% available
for garbage collector).

As your process is using 650megs, and the virtual memory is limited
to 2GB, we're getting close to the boundary. Add some bad luck
with memory fragmentation, and erlang can easily fail to allocate
big chunk of continuous memory.

That's what happens to you.

Possible solutions:
 - upgrade to erlang >= R13B3
   http://www.lshift.net/blog/2009/12/01/garbage-collection-in-erlang
 - use multiple (smaller) queues (erlang GC should be isolated to
   memory used by one erlang process)
 - Configure 'vm_memory_high_watermark' value (as Jerry suggested).
   The smaller the value, the sooner Rabbit will stop accepting
   new messages and growing memory.
 - Switch to 64-bit operating system supported by Erlang.
 - Play with IMAGE_FILE_LARGE_ADDRESS_AWARE flag.

We're looking forward to see a proper 64-bit erlang release for windows,
but that's Ericsson's job.

Cheers,
   Marek