[rabbitmq-discuss] Windows RabbitMQ Crashes and Blue Screens under Load

[james.poole at rsa.com]

We've let loose one of our testing ninjas on RabbitMQ for load testing, and
we're consistently running into issues when the high memory watermark is
hit.

 

Windows Server 2003 32-bit , Erlang R15B 32-bit, Rabbit 2.7.1

 

2,000 Consumers each with their own queue bound to a direct exchange

1 Producer, publishing a 2 MB message to the exchange, once every second,
for a total of 50 seconds

 

Everything behaves as expected, until the memory footprint hits the high
watermark, at which point:

On a physical machine: ERL process crashes and dump file is created

On a Virtual Machine: Blue Screen of Death is shown and server reboots

 

VM environment = VMware, Inc.R vCenter Lab Manager 4.0 (4.0.3.1318)

 

One other note is that we see the same problem with ERL R14B04 and Rabbit
2.7.0.

 

I have looked through the log file and also turned on the console debug
output, and nothing seems to be jumping out as an error.  If needed, I can
upload the minidump from the Blue Screen and the ERL crash dump file, just
point me where to do it.

 

Let me know if there is anything else I can do to try and help get this
fixed.

 

 

 

In the rabbit log, there are no errors, and only a few warnings 20 seconds
before the crash:

 

=INFO REPORT==== 11-Jan-2012::10:55:53 ===

closing TCP connection <0.4405.0> from 10.6.64.104:57830

 

=WARNING REPORT==== 11-Jan-2012::10:55:53 ===

exception on TCP connection <0.20552.0> from 10.6.64.104:59521

connection_closed_abruptly

 

 

In the console output log file for the physical machine, this is the only
message I see:

 

starting direct_client
...done

starting notify cluster nodes
...done

 

broker running

Eshell V5.9  (abort with ^G)

(rabbit at QEDLP082)1> 

Crash dump was written to: C:/Documents and
Settings/Administrator.QEDLP/Application Data/RabbitMQ/erl_crash.dump

eheap_alloc: Cannot allocate 6731340 bytes of memory (of type "heap").

in message_loop

win32sysinfo:Erlang has closed.

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120111/e1621720/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 7172 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120111/e1621720/attachment.bin>