<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Hi,</div><div>since I can not find out the reasons of our problems yet, I've decided to perform some more basics tests.</div><div><br></div><div>Here are the latest results I' ve gathered by doing some test with the MultiCastMain program from java client 2.0.0</div><div><br></div><div>First of all, let me introduce the <b>modifications</b> I have performed on the MultiCastMain to allow more flexibility in the configuration :</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- introduction of optional parameter "<b>-D</b>" to toggle Durability of exchange and queue ON (missing value means OFF)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- introduction of optional parameter "<b>-A</b>" to toggle autodelete of the queue ON (missing value means OFF)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- introduction of optional parameter "<b>-X</b>" to toggle exclusiveness on the queue ON (missing value means OFF)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- introduction of optional parameter "<b>-Q</b>" to give a name to a queue : useful to fetch messages from the same queue between several executions.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- introduction of optional parameter "<b>-K</b>" to give the same routing key (for producer) and binding key (for consumer) between several executions.</div><div><br></div><div>Here after is a snippet of /etc/rabbitmq/rabbitmq.config :</div><div><div><b><span class="Apple-tab-span" style="white-space:pre"> </span>[</b></div><div><b> <span class="Apple-tab-span" style="white-space:pre"> </span>{rabbit, [</b></div><div><b> <span class="Apple-tab-span" style="white-space:pre"> </span>{vm_memory_high_watermark, 0.05},</b></div><div><b> <span class="Apple-tab-span" style="white-space:pre"> </span>{ssl_listeners, [{"0.0.0.0",5671}]},</b></div><div><b> <span class="Apple-tab-span" style="white-space:pre"> </span>{ssl_options, [{cacertfile,"/var/lib/rabbitmq/ssl/certificate-authority/schneider-ca.pem"},</b></div><div><b> <span class="Apple-tab-span" style="white-space:pre"> </span>{certfile,"/var/lib/rabbitmq/ssl/broker-certs/broker-cert.pem"},</b></div><div><b> <span class="Apple-tab-span" style="white-space:pre"> </span>{keyfile,"/var/lib/rabbitmq/ssl/broker-certs/broker-key.pem"},</b></div><div><b> <span class="Apple-tab-span" style="white-space:pre"> </span>{verify,verify_none},</b></div><div><b> <span class="Apple-tab-span" style="white-space:pre"> </span>{fail_if_no_peer_cert,false}]}</b></div><div><b> <span class="Apple-tab-span" style="white-space:pre"> </span>]}<span class="Apple-tab-span" style="white-space:pre"> </span></b></div><div><b><span class="Apple-tab-span" style="white-space:pre"> </span>].</b></div></div><div><br></div><div><div>To step in the memory limitations faster, I 've decreased the memory watermark threshold to .05 (5%) </div></div><div>This corresponds to 50 MB on the host we are using for this test as shown on the rabbit.log file upon startup of broker :</div><div><br></div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span><b>=INFO REPORT==== 15-Sep-2010::12:36:15 ===</b></div><div><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>Limiting to approx 65435 file handles (58889 sockets)</b></div><div><b><br></b></div><div><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>=INFO REPORT==== 15-Sep-2010::12:36:15 ===</b></div><div><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>Memory limit set to 50MB.</b></div></div><div><b><br></b></div><div>Finally the broker status description (rabbitmqctl status)</div><div><br></div><div><div><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>[{running_applications,[{rabbit_status,"RabbitMQ Status Page","0.01"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b> {rabbit,"RabbitMQ","2.0.0"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b> {ssl,"Erlang/OTP SSL application","3.10.7"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{public_key,"Public key infrastructure","0.4"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{mnesia,"MNESIA CXC 138 12","4.4.12"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{os_mon,"CPO CXC 138 46","2.2.4"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{sasl,"SASL CXC 138 11","2.1.8"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{rabbit_mochiweb,"RabbitMQ Mochiweb Embedding","0.01"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{mochiweb,"MochiMedia Web Server","1.3"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{crypto,"CRYPTO version 1","1.6.3"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{inets,"INETS CXC 138 49","5.2"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{stdlib,"ERTS CXC 138 10","1.16.4"},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{kernel,"ERTS CXC 138 10","2.13.4"}]},</b></div><div><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b> {nodes,[{disc,['rabbit@mystic-buntu']}]},</b></div><div><b> </b><span class="Apple-tab-span" style="white-space:pre"><b> </b></span><b>{running_nodes,['rabbit@mystic-buntu']}]</b></div></div><div><br></div><div><div>Thanks to the modifications of the MultiCastMain, I've run a 2 phases producer / consumer test as described bellow :</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span><i>phase 0</i> : initialization of exchange and queue for the test : </div><div><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-style-span" style="font-family: Courier; "><b>MultiCastMain</b></span><span class="Apple-style-span" style="font-family: Courier; "> -h <host> -p 5672 -s 1000 -n 1 -x 0 -y 1 -e </span><span class="Apple-style-span" style="font-family: Courier; "><b>testX</b></span><span class="Apple-style-span" style="font-family: Courier; "> -Q </span><span class="Apple-style-span" style="font-family: Courier; "><b>testQ</b></span><span class="Apple-style-span" style="font-family: Courier; "> -K </span><span class="Apple-style-span" style="font-family: Courier; "><b>testK</b></span><span class="Apple-style-span" style="font-family: Courier; "> -D -z 10</span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>this launches a single consumer creating the exchange <b>testX</b> and queue <b>testQ</b> with durability <b>ON</b>.</div><div><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="white-space: normal; "><span class="Apple-tab-span" style="white-space: pre; "> </span></span><span class="Apple-style-span" style="white-space: normal; ">The queue is bound to the exchange using key <b>testK</b>. </span></span></span></div><div><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="white-space: normal; "></span></span> T</span>he <span class="Apple-style-span" style="white-space: pre; "><span class="Apple-tab-span" style="white-space:pre">t</span>est is told to end after 10 seconds. </span></div><div><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-tab-span" style="white-space:pre"> </span></span></span></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span><i>phase 1</i> - launch a producer alone and watch memory occupation + connections status and queue depth (via rabbit_status plugin for instance)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-style-span" style="font-family: Courier; "><b>MultiCastMain</b></span><span class="Apple-style-span" style="font-family: Courier; "> -h <host> -p 5672 -s 1000 -n 1 -x 1 -y 0 -e </span><span class="Apple-style-span" style="font-family: Courier; "><b>testX</b></span><span class="Apple-style-span" style="font-family: Courier; "> -Q </span><span class="Apple-style-span" style="font-family: Courier; "><b>testQ</b></span><span class="Apple-style-span" style="font-family: Courier; "> -K </span><span class="Apple-style-span" style="font-family: Courier; "><b>testK</b></span><span class="Apple-style-span" style="font-family: Courier; "> -D</span></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><span class="Apple-style-span" style="font-family: Courier; "><div style="font-family: Helvetica; "><span class="Apple-tab-span" style="white-space: pre; "> T</span>h<span class="Apple-style-span" style="white-space: pre;">is run a producer that will publish 1kb messages to the <b>testX </b>exchange with routing key <b>testK.</b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-style-span" style="font-weight: normal;">These messages will be queued inside </span>testQ.</b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><br></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-style-span" style="font-weight: normal;">The producer is blocked after only few seconds running as the memory threshold is reached. The queue depth indicated 12 011 messages.</span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-style-span" style="font-weight: normal;"></span></b></span><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal;">(You can scroll to the end of this mail to see an extract of broker log to illustrate the alarm_hanlder notifications.)</span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal;"><br></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal;">What do I mean by "the producer is blocked" :</span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal;"><span class="Apple-tab-span" style="white-space:pre"> </span>- rabbit_status plugin shows the connection in status "</span>bocked<span class="Apple-style-span" style="font-weight: normal;">"</span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-style-span" style="font-weight: normal;"><span class="Apple-tab-span" style="white-space:pre"> </span>- rabbit_status plugin shows that the depth of queue </span>testQ </b>remains constant</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-tab-span" style="white-space:pre"> </span>- The trace from MultiCastMain does not displays new stats of sending rate after a while</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-tab-span" style="white-space:pre"> </span>- The last lines in the broker log are :</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-style-span" style="font-weight: normal;"><span class="Apple-tab-span" style="white-space: pre; "> </span>=INFO REPORT==== 15-Sep-2010::14:53:44 ===
<span class="Apple-tab-span" style="white-space: pre; "> </span>vm_memory_high_watermark set. Memory used:53650536 allowed:52556595
<span class="Apple-tab-span" style="white-space: pre; "> </span>=INFO REPORT==== 15-Sep-2010::14:53:44 ===
<span class="Apple-tab-span" style="white-space: pre; "> </span>alarm_handler: {set,{vm_memory_high_watermark,[]}}</span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-style-span" style="font-weight: normal;"><br></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-style-span" style="font-weight: normal;">From that point, the producer is blocked without any possible recovery. </span></b></span><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal;">As the flow control is designed in v 2.0.0, I would have expected the producer </span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal;">to be released thanks to sawp of messages to the disk. </span></b></span><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal;">Anyway, my second assumption was that a consumer is needed to release pressure on the queue,</span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal;">so I decided to keep the producer </span></b></span><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal;">f running, and start phase 2 as follow :</span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal;"><span class="Apple-tab-span" style="white-space:pre"> </span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-style-span" style="font-weight: normal;"><span class="Apple-tab-span" style="white-space:pre"> </span><i>phase 2</i> - launch a consumer alone </span></b></span>and watch memory occupation + connections status and queue depth (via rabbit_status plugin for instance)</div><div style="font-family: Helvetica; "><br></div><div style="font-family: Helvetica; "><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><b>MultiCastMain</b></span><span class="Apple-style-span" style="font-family: Courier; "> -h <host> -p 5672 -s 1000 -n 1 -x 0 -y 1 -e </span><span class="Apple-style-span" style="font-family: Courier; "><b>testX</b></span><span class="Apple-style-span" style="font-family: Courier; "> -Q </span><span class="Apple-style-span" style="font-family: Courier; "><b>testQ</b></span><span class="Apple-style-span" style="font-family: Courier; "> -K </span><span class="Apple-style-span" style="font-family: Courier; "><b>testK</b></span><span class="Apple-style-span" style="font-family: Courier; "> -D</span></div></span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div><span class="Apple-style-span" style="font-family: Courier; "><div style="font-family: Helvetica; "><span class="Apple-tab-span" style="white-space: pre; "> T</span>h<span class="Apple-style-span" style="white-space: pre; ">is runs a consumer that will get message out the queue</span><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal; "> </span>testQ.</b></span></div></span></div></span></span></b></span></div><div><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-weight: normal; "><br></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><font class="Apple-style-span" face="Helvetica"><span class="Apple-style-span" style="font-weight: normal;">The messages from </span>testQ <span class="Apple-style-span" style="font-weight: normal;">are removed quite fast too, but the memory occupation does not seem to be impacted at all, as the producer is still blocked.</span></font></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="white-space: normal; "><div style="display: inline !important; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="white-space: normal; "><div style="display: inline !important; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "></span></div></div></span></span></span></div></div></span></span></span><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><font class="Apple-style-span" face="Helvetica"><span class="Apple-style-span" style="font-weight: normal;">From the rabbit_status plugin, the status of the connections remains :</span></font></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><font class="Apple-style-span" face="Helvetica"><span class="Apple-style-span" style="font-weight: normal;"></span></font><span class="Apple-tab-span" style="white-space:pre"> </span>- "blocked" </b><font class="Apple-style-span" face="Helvetica">for the producer</font></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><font class="Apple-style-span" face="Helvetica"></font></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"> </span>- "blocking" </b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-style-span" style="font-family: Helvetica; font-weight: normal; "><b><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><font class="Apple-style-span" face="Helvetica">for the consumer</font></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Helvetica; white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal; white-space: normal; font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="font-family: Courier; "><div style="display: inline !important; "><span class="Apple-style-span" style="white-space: pre; "><b> </b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></span></b></span></div></span></div></span></span></b></span></div></span></div></span><span class="Apple-style-span" style="font-weight: normal;"><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;"><br></span></span></b></span></div>And memory usage is still over the threshold : memory (used/available) = 53MB / 50MB</span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;"><br></span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;">The only way I found out to allow memory to fall back bellow the threshold is by killing manually the connection of the producer.</span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;">Killing the producer process on the client side does not do indeed : </span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;"><br></span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;"><span class="Apple-tab-span" style="white-space:pre"> </span>If I kill the producer process, the connection seems to disappear but the memory remains over the threshold.</span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;"><span class="Apple-tab-span" style="white-space:pre"> </span>The rabbit_status plugin seem to fail after this because I got a strange error report in the broker log. (see the end of this long mail !)</span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;"><span class="Apple-tab-span" style="white-space:pre"> </span></span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;"><span class="Apple-tab-span" style="white-space:pre">B</span>y closing the connection on the broker side, the memory has (not always) fallen below the threshold, back to normal (32 MB / 50 MB).</span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;"> </span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;">I think this behaviour is not expected and maybe that could be due to a bug somewhere, </span>since it si fully reproductible<span class="Apple-style-span" style="font-weight: normal;"> on my configuration.</span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;">The fact that the memory occupation never falls below the threshold after message are removed from the queue is particularly strange and</span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;">unwilling from my point of view. It think that this simple test can points out an issue that would explains the problems that I mentioned in the</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;">previous messages, but not sure about it.</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><br></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;">I am sorry for this quite long message, but I thought that the more details you get, the better.</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><br></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;">best regards,</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><br></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;">Romary.</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"> </span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-tab-span" style="white-space:pre">The following is an extract from the broker log during phase 1</span></span></b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><b><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;"><br></span></span></span></b></span></div><div><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">=</font></span></span><font class="Apple-style-span" face="'Courier New'">INFO REPORT==== 15-Sep-2010::14:53:24 ===
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">accepted TCP connection on 0.0.0.0:5672 from 192.168.0.71:1845
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 15-Sep-2010::14:53:24 ===
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">starting TCP connection <0.16120.0> from 192.168.0.71:1845
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 15-Sep-2010::14:53:42 ===
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">vm_memory_high_watermark set. Memory used:52632808 allowed:52556595
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 15-Sep-2010::14:53:42 ===
</font><font class="Apple-style-span" face="'Courier New'"> <span class="Apple-tab-span" style="white-space:pre"> </span>alarm_handler: {set,{vm_memory_high_watermark,[]}}
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 15-Sep-2010::14:53:43 ===
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">vm_memory_high_watermark clear. Memory used:52355320 allowed:52556595
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 15-Sep-2010::14:53:43 ===</font></span></div><div><span class="Apple-style-span" style="white-space: pre; "><font class="Apple-style-span" face="'Courier New'"> <span class="Apple-tab-span" style="white-space:pre"> </span>alarm_handler: {clear,vm_memory_high_watermark}</font></span></div><div><span class="Apple-style-span" style="white-space: pre;"><font class="Apple-style-span" face="'Courier New'">
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 15-Sep-2010::14:53:44 ===</font></span></div><div><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">vm_memory_high_watermark set. Memory used:53650536 allowed:52556595
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">=INFO REPORT==== 15-Sep-2010::14:53:44 ===
</font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'Courier New'"> </font></span><font class="Apple-style-span" face="'Courier New'">alarm_handler: {set,{vm_memory_high_watermark,[]}}</font></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><br></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre;"><span class="Apple-style-span" style="white-space: normal; font-family: Courier; "><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-tab-span" style="white-space: pre; ">The following is an extract from the broker log during phase </span>2</span></b></span></div><div><font class="Apple-style-span" face="Helvetica"><span class="Apple-style-span" style="white-space: pre;"><font class="Apple-style-span" face="Courier"><b><div><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: normal;"> </span></span></div><div><span class="Apple-style-span" style="font-weight: normal;"><font class="Apple-style-span" face="'Courier New'"><span class="Apple-tab-span" style="white-space:pre"> </span>=INFO REPORT==== 15-Sep-2010::15:21:38 ===</font></span></div><div><span class="Apple-style-span" style="font-weight: normal;"><font class="Apple-style-span" face="'Courier New'"><span class="Apple-tab-span" style="white-space:pre"> </span>accepted TCP connection on 0.0.0.0:5672 from 192.168.0.71:1880</font></span></div><div><span class="Apple-style-span" style="font-weight: normal;"><font class="Apple-style-span" face="'Courier New'"><br></font></span></div><div><span class="Apple-style-span" style="font-weight: normal;"><font class="Apple-style-span" face="'Courier New'"><span class="Apple-tab-span" style="white-space:pre"> </span>=INFO REPORT==== 15-Sep-2010::15:21:38 ===</font></span></div><div><span class="Apple-style-span" style="font-weight: normal;"><font class="Apple-style-span" face="'Courier New'"><span class="Apple-tab-span" style="white-space:pre"> </span>starting TCP connection <0.2101.1> from 192.168.0.71:1880</font></span></div><div><br></div><div><span class="Apple-style-span" style="font-weight: normal; white-space: normal; "><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><span class="Apple-tab-span" style="white-space: pre; "><span class="Apple-tab-span" style="white-space: pre; ">The following is </span>t</span>he error report after shutting down the producer process (client side)</b></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="white-space: pre; "><b><br></b></span></div><div><font class="Apple-style-span" face="Helvetica"><span class="Apple-style-span" style="white-space: pre; "><font class="Apple-style-span" face="Courier"><b></b></font></span><font class="Apple-style-span" face="Courier"><b></b></font></font></div></span></div><div><font class="Apple-style-span" face="'Courier New'"><span class="Apple-style-span" style="font-weight: normal;">ERROR REPORT==== 15-Sep-2010::15:37:53 ===
** Generic server rabbit_status_web terminating
** Last message in was get_context
** When Server state == {state,1284557846311,"Wed, 15 Sep 2010 11:37:26 GMT",
[<<"0.0.0.0:5672 ">>],
[[{pid,<<"<0.16120.0>">>},
{address,<<"192.168.0.58">>},
{port,5672},
{peer_address,<<"192.168.0.71">>},
{peer_port,1845},
{recv_oct,12696041},
{recv_cnt,72079},
{send_oct,357},
{send_cnt,5},
{send_pend,0},
{state,<<"blocked">>},
{channels,1},
{user,<<"guest">>},
{vhost,<<"/">>},
{timeout,0},
{frame_max,131072}]],
[[{vhost,<<"/">>},
{name,<<"testQ">>},
{durable,<<"true">>},
{auto_delete,<<"false">>},
{arguments,<<"[]">>},
{pid,<<"<0.12150.0>">>},
{messages_ready,0},
{messages_unacknowledged,0},
{messages_uncommitted,<<"undefined">>},
{messages,0},
{acks_uncommitted,<<"undefined">>},
{consumers,0},
{transactions,<<"undefined">>},
{memory,6171332}]],
33,65535,55843456,52556595.0,161,1048576}
** Reason for termination ==
** {{badmatch,{error,enotconn}},{gen_server,call,[<0.16120.0>,info,infinity]}}
=ERROR REPORT==== 15-Sep-2010::15:37:53 ===
** Generic server <0.16124.0> terminating
** Last message in was {'EXIT',<0.16123.0>,{badmatch,{error,enotconn}}}
** When Server state == {ch,running,1,<0.16120.0>,<0.16122.0>,undefined,none,
{set,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
1,
{[],[]},
{[],[]},
<<"guest">>,<<"/">>,<<>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
<0.16121.0>,
{state,none,undefined}}
** Reason for termination ==
** {badmatch,{error,enotconn}}</span></font>
</div><div><br></div><div><br></div><div><br></div></b></font></span></font></div></span></span></div></span></div></div><div><div>Le 10 sept. 10 à 15:06, Marek Majkowski a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Romary,<br><br>First, thank a lot for the feedback. For example the information about<br>SSL memory<br>usage is indeed very interesting. (if that is a big problem to you,<br>you may always<br>fall back to the old technique of using stunnel)<br><br>The flow control was heavily modified between 1.8.1 and 2.0.0. In summary:<br> - 1.8.1 - we have send Channel.flow AMQP message to everyone once<br> rabbit reached memory limit<br> - 2.0.0 - once we reach memory limit, the connections from which we hear<br> publishes are stopped temporarily. We stop receiving bytes from tcp sockets.<br> That 'stop' shouldn't take too long, as data should be swapped out to disk<br> and memory pressure will drop pretty quickly.<br><br><br>On Wed, Sep 8, 2010 at 11:49, Romary Kremer <<a href="mailto:romary.kremer@gmail.com">romary.kremer@gmail.com</a>> wrote:<br><blockquote type="cite">I've started playing a bit with the latest release 2.0.0 and I m affraid<br></blockquote><blockquote type="cite">that it looks like their are some regression or at least some semantic<br></blockquote><blockquote type="cite">updates.<br></blockquote><br>It's best if you upgraded both server and client library. Do you have any<br>particular problems? A lot was changed in 2.0.0 but we think it's fully<br>functional. If you found something that blocks you to migrate, you<br>could report a bug.<br><br><blockquote type="cite">It looks like any listener is called back when the alarm handler is set or<br></blockquote><blockquote type="cite">cleared, while the producers are still paused / resumed<br></blockquote><blockquote type="cite">like their are to be.<br></blockquote><br>Interesting. Maybe we have a race there? Or maybe you're blocking<br>the main java client thread? (nothing blocking should be done from<br>the main thread)<br><br><blockquote type="cite"><blockquote type="cite">during long running tests, we have encountered strange behaviour due to<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">flow control :<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">The queue depth starts to increase linearly for about 2 hours, these is<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">coherent since the message throughput of the single consumer<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">is not enough to absorb message ingress. Memory occupation grow faster as<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">well, until the memory watermark is reached on the broker side.<br></blockquote></blockquote><br>Are you sure your consumer is ACK-ing the messages it received?<br><br><blockquote type="cite"><blockquote type="cite">From that point, the producers are indeed paused, as flow control request<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">has been issued by the broker, but the consumer seems to be blocked<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">as well. The queue level is flatten at its top value until the end of the<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">test, even when memory occupation lowered under the threshold.<br></blockquote></blockquote><br>That's how 1.8.1 behaves. In 2.0.0 we introduced swapping out big queues<br>to disk, so the memory usage shouldn't be dependent on a queue size.<br><br><blockquote type="cite"><blockquote type="cite">By registering the FlowListener callback, we have noticed that not all of<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">the producers are notified all the time the alarm handler is set.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Does this mean that the broker applies some heuristic to try not to block<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">every body every time ?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Or does it mean that some of the channels have been somehow blacklisted by<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">the broker ?<br></blockquote></blockquote><br>No, in 1.8.1 broker should send 'channel.flow' to all the channels.<br><br><blockquote type="cite"><blockquote type="cite">Could anybody explain how the blocking of consumer is assumed to be<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">implemented ?<br></blockquote></blockquote><br>The best description is probably here:<br> <a href="http://www.rabbitmq.com/extensions.html#memsup">http://www.rabbitmq.com/extensions.html#memsup</a><br><br>But it covers 2.0.0. I'd suggest an upgrade to 2.0.0 and monitoring<br>not only queue size but also number of unacknowledged messages<br>('Msg unack' in status plugin). This number should be near zero.<br><br>Cheers,<br> Marek Majkowski<br></div></blockquote></div><br></body></html>