[rabbitmq-discuss] ERL crashes when messages are being pumped into a queue under win32 envir

[Jerry Kuch]

> My test environment: receiver and broker running on the same hardware,
> the broker ran out of memory and not the receiver, since we do not
> process any messages received.
>
> RabbitMQ's RAM started to increase and fast, far exceeded win32's
> limitation.  It does crash at times, but other times, my receiver and
> senders would stop working...

Rabbit uses a memory-based flow control, and TCP back pressure to try
to protect a broker where publications are vastly outstripping
consumption, as Emile described.  Basically, it stops reading on the
producer sockets, until it has a change to reclaim some memory by
pushing queued messages out to disk.

That said, the mechanism is possible to overwhelm, particularly on
32-bit machines, and Rabbit on Windows seems especially sensitive.
Unfortunately Ericsson doesn't distribute a 64-bit version of Erlang
for Windows, so the easy way out, running in an environment with a
larger user address space, isn't available.

> RabbitMQ would still being shown in the
> process list, but when use rabbitmqctl to check the state of it, it
> tells me the node is already down/none existent.

The Rabbit process still being alive, but rabbitmqctl thinking it's
down is, I think a strange behavior.

> =INFO REPORT==== 1-Apr-2011::09:37:20 ===
> vm_memory_high_watermark set. Memory used:860143728 allowed:858993459
>
> =INFO REPORT==== 1-Apr-2011::09:37:20 ===
>     alarm_handler: {set,{{vm_memory_high_watermark,'rabbit at DEV-004'},[]}}
>
> =INFO REPORT==== 1-Apr-2011::09:37:20 ===
> vm_memory_high_watermark clear. Memory used:858369128 allowed:858993459
>
> =INFO REPORT==== 1-Apr-2011::09:37:20 ===
>     alarm_handler: {clear,{vm_memory_high_watermark,'rabbit at DEV-004'}}
>
> =INFO REPORT==== 1-Apr-2011::09:37:22 ===
> vm_memory_high_watermark set. Memory used:859860800 allowed:858993459
>
> =INFO REPORT==== 1-Apr-2011::09:37:22 ===
>     alarm_handler: {set,{{vm_memory_high_watermark,'rabbit at DEV-004'},[]}}

Your log messages are what one would expect if memory-based flow
control is kicking in.  Since you still seem to be seeing a Rabbit
process, just a zombie-ish one, I'd imagine you're not getting an
erl_crash.dump file?  Or are you?  If you find one, the Slogan near
the top is usually pretty explicit if an allocation failure happens
that topples the Erlang VM when under memory pressure.

Jerry