<font face="trebuchet ms,sans-serif">Are you acking messages? If you don't, those messages will have to be stored somewhere and 2000 * 2MB * 1/s = 4 GB/s<br></font><br><div class="gmail_quote">On Thu, Jan 12, 2012 at 18:55, <span dir="ltr"><<a href="mailto:james.poole@rsa.com">james.poole@rsa.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Simone, that would be great if you could try to reproduce it.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u>�<u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">As mentioned, we are creating 2000 consumers each with their own queue bound to a fanout exchange.� After the queues have all been created and bound, a producer publishes a 2 MB message to this fanout exchange once every second for 50 seconds.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u>�<u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">All queues are non-durable.� And autoAck was set to false in the Java client.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u>�<u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Everything hums along until the vm_memory_high_watermark is triggered and then we see the crash. �One interesting thing is that in the log it still shows it accepting and starting tcp connections after the memory alarm is triggered (for around 15 seconds before the crash).� I thought this was supposed to block until the memory was under control?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u>�<u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-James<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u>�<u></u></span></p><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Simone Busoli [mailto:<a href="mailto:simone.busoli@gmail.com" target="_blank">simone.busoli@gmail.com</a>] <br>
<b>Sent:</b> Wednesday, January 11, 2012 3:02 PM<br><b>To:</b> Poole, James<br><b>Cc:</b> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a>; Kuch, Jerry (VMware)</span></p>
<div><div class="h5"><br><b>Subject:</b> Re: [rabbitmq-discuss] Windows RabbitMQ Crashes and Blue Screens under Load<u></u><u></u></div></div><p></p></div><div><div class="h5"><p class="MsoNormal"><u></u>�<u></u></p><p>Hi James,<u></u><u></u></p>
<p>If you can provide more details about the load you're applying to the broker I would be glad to try to reproduce it.<br>We've been using RabbitMQ on Windows in production for some months now and didn't experience any weird behavior.<br>
What I'm interested in is whether entities and messages are durable, if you use transactions or publisher confirms and the like.<u></u><u></u></p><div><p class="MsoNormal">On Jan 11, 2012 7:52 PM, <<a href="mailto:james.poole@rsa.com" target="_blank">james.poole@rsa.com</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Yeah, I should have mentioned that we started out testing with the 64-bit version and found this issue... though the VM probably didn't have very much more memory than a 32-bit address space would provide. �Then we backed down to the 32-bit version to see if it went away, but it didn't.<br>
<br>I will see if we can send out the test program (it's just a simple java app using the rabbitmq-java-client-2.7.1). �If I can send it out, how would I go about this... attach to the email or upload it to a server somewhere?<br>
<br>-James<br><br>-----Original Message-----<br>From: Jerry Kuch [mailto:<a href="mailto:jerryk@vmware.com" target="_blank">jerryk@vmware.com</a>]<br>Sent: Wednesday, January 11, 2012 1:44 PM<br>To: Poole, James<br>Cc: <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
Subject: Re: [rabbitmq-discuss] Windows RabbitMQ Crashes and Blue Screens under Load<br><br>James: �Out of curiousity have you tried the new 64-bit release of<br>Erlang for Windows in your environment? �The address space size<br>
limitations of the 32-bit version have been associated with crashy<br>Rabbits in the past (although bringing your memory high watermark<br>value down so that the back-pressure mechanisms engage when the<br>broker is in less trouble may help). �I think you can scare up the<br>
new Erlang here:<br><br><a href="http://www.erlang.org/download/otp_win64_R15B.exe" target="_blank">http://www.erlang.org/download/otp_win64_R15B.exe</a><br><br>Until recently there was no 64-bit Erlang, so even those running on<br>
64-bit Windows boxes were still relegated to 32-bit VMs.<br><br>I am curious about the different results between a physical machine<br>and a virtualized one, with one showing a "clean" Erlang VM crash and<br>the other exhibiting a blue-screen, fatal OS-wrecker...<br>
<br>Is the traffic you're using to bring these systems down part of a<br>large or proprietary app, or can you extract a bare minimum piece<br>of code that brings the pain and share it with us? �If you could<br>do the latter we could more easily investigate the situation within<br>
VMware since the difference in behavior between baremetal and<br>virtualization is disquieting...<br><br>Best regards,<br>Jerry<br><br>----- Original Message -----<br>From: "james poole" <<a href="mailto:james.poole@rsa.com" target="_blank">james.poole@rsa.com</a>><br>
To: <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>Sent: Wednesday, January 11, 2012 10:32:23 AM<br>Subject: [rabbitmq-discuss] Windows RabbitMQ Crashes and Blue Screens under � � Load<br>
<br><br><br><br><br>We�ve let loose one of our testing ninjas on RabbitMQ for load testing, and we�re consistently running into issues when the high memory watermark is hit.<br><br><br><br>Windows Server 2003 32-bit , Erlang R15B 32-bit, Rabbit 2.7.1<br>
<br><br><br>2,000 Consumers each with their own queue bound to a direct exchange<br><br>1 Producer, publishing a 2 MB message to the exchange, once every second, for a total of 50 seconds<br><br><br><br>Everything behaves as expected, until the memory footprint hits the high watermark, at which point:<br>
<br>On a physical machine: ERL process crashes and dump file is created<br><br>On a Virtual Machine: Blue Screen of Death is shown and server reboots<br><br><br><br>VM environment = VMware, Inc.� vCenter Lab Manager 4.0 (4.0.3.1318)<br>
<br><br><br>One other note is that we see the same problem with ERL R14B04 and Rabbit 2.7.0.<br><br><br><br>I have looked through the log file and also turned on the console debug output, and nothing seems to be jumping out as an error. If needed, I can upload the minidump from the Blue Screen and the ERL crash dump file, just point me where to do it.<br>
<br><br><br>Let me know if there is anything else I can do to try and help get this fixed.<br><br><br><br><br><br><br><br>In the rabbit log, there are no errors, and only a few warnings 20 seconds before the crash:<br><br>
<br><br>=INFO REPORT==== 11-Jan-2012::10:55:53 ===<br><br>closing TCP connection <0.4405.0> from <a href="http://10.6.64.104:57830" target="_blank">10.6.64.104:57830</a><br><br><br><br>=WARNING REPORT==== 11-Jan-2012::10:55:53 ===<br>
<br>exception on TCP connection <0.20552.0> from <a href="http://10.6.64.104:59521" target="_blank">10.6.64.104:59521</a><br><br>connection_closed_abruptly<br><br><br><br><br><br>In the console output log file for the physical machine, this is the only message I see:<br>
<br><br><br>starting direct_client ...done<br><br>starting notify cluster nodes ...done<br><br><br><br>broker running<br><br>Eshell V5.9 (abort with ^G)<br><br>(rabbit@QEDLP082)1><br><br>Crash dump was written to: C:/Documents and Settings/Administrator.QEDLP/Application Data/RabbitMQ/erl_crash.dump<br>
<br>eheap_alloc: Cannot allocate 6731340 bytes of memory (of type "heap").<br><br>in message_loop<br><br>win32sysinfo:Erlang has closed.<br><br><br><br><br>_______________________________________________<br>rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><u></u><u></u></p>
</div></div></div></div></div></blockquote></div><br>