<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Trebuchet MS";
panose-1:2 11 6 3 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>We are sending the Ack from the consumer side after the message is delivered. I will try setting autoAck to true and see if that makes a difference.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I thought Rabbit was smart enough to just keep one copy of a message for a fanout? Does it really create 2000 separate copies of the same message in memory, or does it just do this when the message is being sent?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-James<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Simone Busoli [mailto:simone.busoli@gmail.com] <br><b>Sent:</b> Thursday, January 12, 2012 1:03 PM<br><b>To:</b> Poole, James<br><b>Cc:</b> rabbitmq-discuss@lists.rabbitmq.com; Kuch, Jerry (VMware)<br><b>Subject:</b> Re: [rabbitmq-discuss] Windows RabbitMQ Crashes and Blue Screens under Load<o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:"Trebuchet MS","sans-serif"'>Are you acking messages? If you don't, those messages will have to be stored somewhere and 2000 * 2MB * 1/s = 4 GB/s</span><o:p></o:p></p><div><p class=MsoNormal>On Thu, Jan 12, 2012 at 18:55, <<a href="mailto:james.poole@rsa.com">james.poole@rsa.com</a>> wrote:<o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Simone, that would be great if you could try to reproduce it.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>As mentioned, we are creating 2000 consumers each with their own queue bound to a fanout exchange. After the queues have all been created and bound, a producer publishes a 2 MB message to this fanout exchange once every second for 50 seconds.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>All queues are non-durable. And autoAck was set to false in the Java client.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Everything hums along until the vm_memory_high_watermark is triggered and then we see the crash. One interesting thing is that in the log it still shows it accepting and starting tcp connections after the memory alarm is triggered (for around 15 seconds before the crash). I thought this was supposed to block until the memory was under control?</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-James</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Simone Busoli [mailto:<a href="mailto:simone.busoli@gmail.com" target="_blank">simone.busoli@gmail.com</a>] <br><b>Sent:</b> Wednesday, January 11, 2012 3:02 PM<br><b>To:</b> Poole, James<br><b>Cc:</b> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a>; Kuch, Jerry (VMware)</span><o:p></o:p></p><div><div><p class=MsoNormal><br><b>Subject:</b> Re: [rabbitmq-discuss] Windows RabbitMQ Crashes and Blue Screens under Load<o:p></o:p></p></div></div></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p>Hi James,<o:p></o:p></p><p>If you can provide more details about the load you're applying to the broker I would be glad to try to reproduce it.<br>We've been using RabbitMQ on Windows in production for some months now and didn't experience any weird behavior.<br>What I'm interested in is whether entities and messages are durable, if you use transactions or publisher confirms and the like.<o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Jan 11, 2012 7:52 PM, <<a href="mailto:james.poole@rsa.com" target="_blank">james.poole@rsa.com</a>> wrote:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'>Yeah, I should have mentioned that we started out testing with the 64-bit version and found this issue... though the VM probably didn't have very much more memory than a 32-bit address space would provide. Then we backed down to the 32-bit version to see if it went away, but it didn't.<br><br>I will see if we can send out the test program (it's just a simple java app using the rabbitmq-java-client-2.7.1). If I can send it out, how would I go about this... attach to the email or upload it to a server somewhere?<br><br>-James<br><br>-----Original Message-----<br>From: Jerry Kuch [mailto:<a href="mailto:jerryk@vmware.com" target="_blank">jerryk@vmware.com</a>]<br>Sent: Wednesday, January 11, 2012 1:44 PM<br>To: Poole, James<br>Cc: <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>Subject: Re: [rabbitmq-discuss] Windows RabbitMQ Crashes and Blue Screens under Load<br><br>James: Out of curiousity have you tried the new 64-bit release of<br>Erlang for Windows in your environment? The address space size<br>limitations of the 32-bit version have been associated with crashy<br>Rabbits in the past (although bringing your memory high watermark<br>value down so that the back-pressure mechanisms engage when the<br>broker is in less trouble may help). I think you can scare up the<br>new Erlang here:<br><br><a href="http://www.erlang.org/download/otp_win64_R15B.exe" target="_blank">http://www.erlang.org/download/otp_win64_R15B.exe</a><br><br>Until recently there was no 64-bit Erlang, so even those running on<br>64-bit Windows boxes were still relegated to 32-bit VMs.<br><br>I am curious about the different results between a physical machine<br>and a virtualized one, with one showing a "clean" Erlang VM crash and<br>the other exhibiting a blue-screen, fatal OS-wrecker...<br><br>Is the traffic you're using to bring these systems down part of a<br>large or proprietary app, or can you extract a bare minimum piece<br>of code that brings the pain and share it with us? If you could<br>do the latter we could more easily investigate the situation within<br>VMware since the difference in behavior between baremetal and<br>virtualization is disquieting...<br><br>Best regards,<br>Jerry<br><br>----- Original Message -----<br>From: "james poole" <<a href="mailto:james.poole@rsa.com" target="_blank">james.poole@rsa.com</a>><br>To: <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>Sent: Wednesday, January 11, 2012 10:32:23 AM<br>Subject: [rabbitmq-discuss] Windows RabbitMQ Crashes and Blue Screens under Load<br><br><br><br><br><br>We’ve let loose one of our testing ninjas on RabbitMQ for load testing, and we’re consistently running into issues when the high memory watermark is hit.<br><br><br><br>Windows Server 2003 32-bit , Erlang R15B 32-bit, Rabbit 2.7.1<br><br><br><br>2,000 Consumers each with their own queue bound to a direct exchange<br><br>1 Producer, publishing a 2 MB message to the exchange, once every second, for a total of 50 seconds<br><br><br><br>Everything behaves as expected, until the memory footprint hits the high watermark, at which point:<br><br>On a physical machine: ERL process crashes and dump file is created<br><br>On a Virtual Machine: Blue Screen of Death is shown and server reboots<br><br><br><br>VM environment = VMware, Inc.® vCenter Lab Manager 4.0 (4.0.3.1318)<br><br><br><br>One other note is that we see the same problem with ERL R14B04 and Rabbit 2.7.0.<br><br><br><br>I have looked through the log file and also turned on the console debug output, and nothing seems to be jumping out as an error. If needed, I can upload the minidump from the Blue Screen and the ERL crash dump file, just point me where to do it.<br><br><br><br>Let me know if there is anything else I can do to try and help get this fixed.<br><br><br><br><br><br><br><br>In the rabbit log, there are no errors, and only a few warnings 20 seconds before the crash:<br><br><br><br>=INFO REPORT==== 11-Jan-2012::10:55:53 ===<br><br>closing TCP connection <0.4405.0> from <a href="http://10.6.64.104:57830" target="_blank">10.6.64.104:57830</a><br><br><br><br>=WARNING REPORT==== 11-Jan-2012::10:55:53 ===<br><br>exception on TCP connection <0.20552.0> from <a href="http://10.6.64.104:59521" target="_blank">10.6.64.104:59521</a><br><br>connection_closed_abruptly<br><br><br><br><br><br>In the console output log file for the physical machine, this is the only message I see:<br><br><br><br>starting direct_client ...done<br><br>starting notify cluster nodes ...done<br><br><br><br>broker running<br><br>Eshell V5.9 (abort with ^G)<br><br>(rabbit@QEDLP082)1><br><br>Crash dump was written to: C:/Documents and Settings/Administrator.QEDLP/Application Data/RabbitMQ/erl_crash.dump<br><br>eheap_alloc: Cannot allocate 6731340 bytes of memory (of type "heap").<br><br>in message_loop<br><br>win32sysinfo:Erlang has closed.<br><br><br><br><br>_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br><br>_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><o:p></o:p></p></div></div></div></div></div></div><p class=MsoNormal><o:p> </o:p></p></div></body></html>