<p>Hi James,</p>
<p>If you can provide more details about the load you're applying to the broker I would be glad to try to reproduce it.<br>
We've been using RabbitMQ on Windows in production for some months now and didn't experience any weird behavior.<br>
What I'm interested in is whether entities and messages are durable, if you use transactions or publisher confirms and the like.</p>
<div class="gmail_quote">On Jan 11, 2012 7:52 PM, <<a href="mailto:james.poole@rsa.com">james.poole@rsa.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Yeah, I should have mentioned that we started out testing with the 64-bit version and found this issue... though the VM probably didn't have very much more memory than a 32-bit address space would provide. Then we backed down to the 32-bit version to see if it went away, but it didn't.<br>
<br>
I will see if we can send out the test program (it's just a simple java app using the rabbitmq-java-client-2.7.1). If I can send it out, how would I go about this... attach to the email or upload it to a server somewhere?<br>
<br>
-James<br>
<br>
-----Original Message-----<br>
From: Jerry Kuch [mailto:<a href="mailto:jerryk@vmware.com">jerryk@vmware.com</a>]<br>
Sent: Wednesday, January 11, 2012 1:44 PM<br>
To: Poole, James<br>
Cc: <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
Subject: Re: [rabbitmq-discuss] Windows RabbitMQ Crashes and Blue Screens under Load<br>
<br>
James: Out of curiousity have you tried the new 64-bit release of<br>
Erlang for Windows in your environment? The address space size<br>
limitations of the 32-bit version have been associated with crashy<br>
Rabbits in the past (although bringing your memory high watermark<br>
value down so that the back-pressure mechanisms engage when the<br>
broker is in less trouble may help). I think you can scare up the<br>
new Erlang here:<br>
<br>
<a href="http://www.erlang.org/download/otp_win64_R15B.exe" target="_blank">http://www.erlang.org/download/otp_win64_R15B.exe</a><br>
<br>
Until recently there was no 64-bit Erlang, so even those running on<br>
64-bit Windows boxes were still relegated to 32-bit VMs.<br>
<br>
I am curious about the different results between a physical machine<br>
and a virtualized one, with one showing a "clean" Erlang VM crash and<br>
the other exhibiting a blue-screen, fatal OS-wrecker...<br>
<br>
Is the traffic you're using to bring these systems down part of a<br>
large or proprietary app, or can you extract a bare minimum piece<br>
of code that brings the pain and share it with us? If you could<br>
do the latter we could more easily investigate the situation within<br>
VMware since the difference in behavior between baremetal and<br>
virtualization is disquieting...<br>
<br>
Best regards,<br>
Jerry<br>
<br>
----- Original Message -----<br>
From: "james poole" <<a href="mailto:james.poole@rsa.com">james.poole@rsa.com</a>><br>
To: <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
Sent: Wednesday, January 11, 2012 10:32:23 AM<br>
Subject: [rabbitmq-discuss] Windows RabbitMQ Crashes and Blue Screens under Load<br>
<br>
<br>
<br>
<br>
<br>
We’ve let loose one of our testing ninjas on RabbitMQ for load testing, and we’re consistently running into issues when the high memory watermark is hit.<br>
<br>
<br>
<br>
Windows Server 2003 32-bit , Erlang R15B 32-bit, Rabbit 2.7.1<br>
<br>
<br>
<br>
2,000 Consumers each with their own queue bound to a direct exchange<br>
<br>
1 Producer, publishing a 2 MB message to the exchange, once every second, for a total of 50 seconds<br>
<br>
<br>
<br>
Everything behaves as expected, until the memory footprint hits the high watermark, at which point:<br>
<br>
On a physical machine: ERL process crashes and dump file is created<br>
<br>
On a Virtual Machine: Blue Screen of Death is shown and server reboots<br>
<br>
<br>
<br>
VM environment = VMware, Inc.® vCenter Lab Manager 4.0 (4.0.3.1318)<br>
<br>
<br>
<br>
One other note is that we see the same problem with ERL R14B04 and Rabbit 2.7.0.<br>
<br>
<br>
<br>
I have looked through the log file and also turned on the console debug output, and nothing seems to be jumping out as an error. If needed, I can upload the minidump from the Blue Screen and the ERL crash dump file, just point me where to do it.<br>
<br>
<br>
<br>
Let me know if there is anything else I can do to try and help get this fixed.<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
In the rabbit log, there are no errors, and only a few warnings 20 seconds before the crash:<br>
<br>
<br>
<br>
=INFO REPORT==== 11-Jan-2012::10:55:53 ===<br>
<br>
closing TCP connection <0.4405.0> from <a href="http://10.6.64.104:57830" target="_blank">10.6.64.104:57830</a><br>
<br>
<br>
<br>
=WARNING REPORT==== 11-Jan-2012::10:55:53 ===<br>
<br>
exception on TCP connection <0.20552.0> from <a href="http://10.6.64.104:59521" target="_blank">10.6.64.104:59521</a><br>
<br>
connection_closed_abruptly<br>
<br>
<br>
<br>
<br>
<br>
In the console output log file for the physical machine, this is the only message I see:<br>
<br>
<br>
<br>
starting direct_client ...done<br>
<br>
starting notify cluster nodes ...done<br>
<br>
<br>
<br>
broker running<br>
<br>
Eshell V5.9 (abort with ^G)<br>
<br>
(rabbit@QEDLP082)1><br>
<br>
Crash dump was written to: C:/Documents and Settings/Administrator.QEDLP/Application Data/RabbitMQ/erl_crash.dump<br>
<br>
eheap_alloc: Cannot allocate 6731340 bytes of memory (of type "heap").<br>
<br>
in message_loop<br>
<br>
win32sysinfo:Erlang has closed.<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div>