<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>Le 16 sept. 10 à 14:12, Emile Joubert a écrit :</div><div><br></div><div>Hi, </div><div>thanks for the support. The same problem happens both with the version 2.0.0 and 2.1.0 !!</div><div>I have just performed the upgrade on our development environment, but it does not seem to solve this issue.</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><br>Hi Romary,<br><br>Thanks for this error report. Can you confirm whether the logfile<br>contained anything relevant before the broker became unavailable?</div></blockquote><div><br></div><div>Nope, the broker log does not contain anything relevant. Here is an extract of the log file when the broker crashes :</div><div><br></div><div><div><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 16-Sep-2010::14:47:47 ===</font></div><div><font class="Apple-style-span" face="'Courier New'">Limiting to approx 65435 file handles (58889 sockets)</font></div><div><font class="Apple-style-span" face="'Courier New'"><br></font></div><div><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 16-Sep-2010::14:47:47 ===</font></div><div><font class="Apple-style-span" face="'Courier New'">Memory limit set to 1609MB.</font></div><div><font class="Apple-style-span" face="'Courier New'"><br></font></div><div><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 16-Sep-2010::14:47:47 ===</font></div><div><font class="Apple-style-span" face="'Courier New'">msg_store_transient: using rabbit_msg_store_ets_index to provide index</font></div><div><font class="Apple-style-span" face="'Courier New'"><br></font></div><div><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 16-Sep-2010::14:47:47 ===</font></div><div><font class="Apple-style-span" face="'Courier New'">msg_store_persistent: using rabbit_msg_store_ets_index to provide index</font></div><div><font class="Apple-style-span" face="'Courier New'"><br></font></div><div><font class="Apple-style-span" face="'Courier New'">=WARNING REPORT==== 16-Sep-2010::14:47:47 ===</font></div><div><font class="Apple-style-span" face="'Courier New'">msg_store_persistent: rebuilding indices from scratch</font></div><div><font class="Apple-style-span" face="'Courier New'"><br></font></div><div><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 16-Sep-2010::14:47:47 ===</font></div><div><font class="Apple-style-span" face="'Courier New'">started TCP Listener on 0.0.0.0:5672</font></div><div><font class="Apple-style-span" face="'Courier New'"><br></font></div><div><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 16-Sep-2010::14:47:47 ===</font></div><div><font class="Apple-style-span" face="'Courier New'">started SSL Listener on 0.0.0.0:5671</font></div><div><br></div></div><blockquote type="cite"><div> Do you<br>override any default configuration variables in the broker config file?<br><br></div></blockquote><div><br></div><div>Ho yes we did : </div><div><br></div><div><b>here is the /etc/rabbitmq/rabbitmq.config</b></div><div><b><br></b></div><div><div>[</div><div> {rabbit, [</div><div> {vm_memory_high_watermark, 0.8},</div><div> {ssl_listeners, [{"0.0.0.0",5671}]},</div><div> {ssl_options, [{cacertfile,"/var/lib/rabbitmq/ssl/certificate-authority/schneider-ca.pem"},</div><div> {certfile,"/var/lib/rabbitmq/ssl/broker-certs/broker-cert.pem"},</div><div> {keyfile,"/var/lib/rabbitmq/ssl/broker-certs/broker-key.pem"},</div><div> {verify,verify_none},</div><div> {fail_if_no_peer_cert,false}]}</div><div> ]}</div><div>].</div><div><br></div></div><div><br></div><div><b>and ther is the /etc/rabbitmq/rabbitmq.conf </b></div><div><b><br></b></div><div><div># increase the max number of Erlang lightweight process</div><div>SERVER_ERL_ARGS="$SERVER_ERL_ARGS +P 1000000"</div><div>RABBITMQ_MULTI_ERL_ARGS="$RABBITMQ_MULTI_ERL_ARGS +P 1000000"</div><div><br></div></div><div><b>finaly the /etc/default/rabbitmq</b></div><div><b><br></b></div><div><div># 65,635 maximum connections per rabbitmq process</div><div>ulimit -n 65535</div><div><br></div></div><br><blockquote type="cite"><div>If you find memory usage growing with SSL then try using Erlang R14B.<br>(Also see OTP-8810) What version of Erlang are you currently using?<br></div></blockquote><div><br></div><div>Yes we are on our way to upgrade to Erlang R14B too !! I will let you know if it solves any of our troubles !</div><br><blockquote type="cite"><div>Regards<br><br>Emile<br><br></div></blockquote><div><br></div>Thanks for all,</div><div><br></div><div>Romary.</div><div><br><blockquote type="cite"><div><br><br><br><br>On 16/09/10 10:32, Romary Kremer wrote:<br><blockquote type="cite">We are performing evaluation on rabbitmq message broker, and we<br></blockquote><blockquote type="cite">currently encounter difficulties with release 2.0.0:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">- Our application implies 10 000 peers producing messages periodically<br></blockquote><blockquote type="cite">to a unique queue. This queue is listen asynchronously by another peer.<br></blockquote><blockquote type="cite">- All peer are written in Java.<br></blockquote><blockquote type="cite">- The production rate of a single peer is 4 messages / hour.<br></blockquote><blockquote type="cite">- We can simulate a time-consuming task in the consumer callback,<br></blockquote><blockquote type="cite">simulating more or less fast consumer.<br></blockquote><blockquote type="cite">- we are using SSL certificate on the broker side to allow the peer to<br></blockquote><blockquote type="cite">authenticate the broker.<br></blockquote><blockquote type="cite">- we have noticed that the use of SSL as dramatic incidence on the<br></blockquote><blockquote type="cite">memory occupied by Rabbitmq process<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Since we upgraded to version *2.0.0*, we are no longer able to make a<br></blockquote><blockquote type="cite">test scenario running. The symptoms are listed bellow :<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">on the broker console first, we get the message :<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Erlang has closed<br></blockquote><blockquote type="cite">Error: unable to connect to node rabbit@murphys: nodedown<br></blockquote><blockquote type="cite">diagnostics:<br></blockquote><blockquote type="cite">- nodes and their ports on murphys: [{rabbitmqctl22609,42767}]<br></blockquote><blockquote type="cite">- current node: rabbitmqctl22609@murphys<br></blockquote><blockquote type="cite">- current node home dir: /var/lib/rabbitmq<br></blockquote><blockquote type="cite">- current node cookie hash: qu0gh1hg7j7LKyzK0GLk+A==<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">we have kept the erl_crash.dump in case, but since i's about 200 MB I<br></blockquote><blockquote type="cite">cannot do nothing to send it to you.<br></blockquote><blockquote type="cite">Maybe some one can give us some hints or some indicators to look out in<br></blockquote><blockquote type="cite">the dump to help diagnostics, but we are not Erlang fluent !<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">What we know for sure is that the crash happens while the 10 000<br></blockquote><blockquote type="cite">connections are established, at the beginning of the test. <br></blockquote><blockquote type="cite">We have monitored the number of connections established and the crashes<br></blockquote><blockquote type="cite">happens always around 4500 - 5000 connections, but never the same exact<br></blockquote><blockquote type="cite">number. <br></blockquote><blockquote type="cite">We also tried with and without SSL but this does not help at all (same<br></blockquote><blockquote type="cite">symptoms).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On the client side, our application registers a ShutdownListener to<br></blockquote><blockquote type="cite">implement a connection retry logic upon shutdown. <br></blockquote><blockquote type="cite">The retries always failed with the error : connection refused.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">here are some figures we gathered during the test start up about the<br></blockquote><blockquote type="cite">maximum number of connection established before it crashes <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">- with SSL : 5404, 4493, 4399<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">- without SSL : 4673<br></blockquote><blockquote type="cite">we dont think that the problem is about file descriptors since we<br></blockquote><blockquote type="cite">haven't changed anything in the configuration when we upgraded to 2.0.0.<br></blockquote><blockquote type="cite">The same test used to run successfully on previous version of the broker<br></blockquote><blockquote type="cite">(1.7.2, and 1.8.1).<br></blockquote><blockquote type="cite">Moreover, the rabbit_status plugin tells us we have enough file<br></blockquote><blockquote type="cite">descriptors as well as erlang processes<br></blockquote><blockquote type="cite">- file descriptors (used / available)= 34 / 65535<br></blockquote><blockquote type="cite">- elrang processes (used / available)= 160 / 1 000 000<br></blockquote><blockquote type="cite">- memory (used / available)= 40 MB / 1609 MB<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">We haven't try the *2.1.0* yet because we would like to have your<br></blockquote><blockquote type="cite">feedback about this issue before.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">We would appreciate your feedbacks on that point before we migrate to<br></blockquote><blockquote type="cite">release 2.1.0.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Best regards,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Romary.<br></blockquote></div></blockquote></div><br></body></html>