Just pinging this thread as it seems to have gone dead, and it feels like a serious issue with clustering/HA in 2.8.6.<br><br>When we last left off, Matthias had asked for the logs for the two other brokers when the queue disappeared. My last message contained the contents of those logs.<br>
<br>Thanks,<br><br>Matt<br><br><div class="gmail_quote">On Fri, Aug 24, 2012 at 11:22 AM, Matt Pietrek <span dir="ltr"><<a href="mailto:mpietrek@skytap.com" target="_blank">mpietrek@skytap.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">> The logs for play and util from around the same time might offer us some more clues.</div><span><font color="#888888"><br>
<br>What I sent was rabbit@play.log.<br><br>rabbit@play2.log doesn't look nearly as interesting:<div class="im"><br>
<br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>Limiting to approx 924 file handles (829 sockets)<div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>Memory limit set to 399MB of 999MB total.<div class="im">
<br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>
Disk free limit set to 100MB<div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>Management plugin upgraded statistics to fine.<div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br>
</div>msg_store_transient: using rabbit_msg_store_ets_index to provide index<div class="im"><br>
<br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>msg_store_persistent: using rabbit_msg_store_ets_index to provide index<br><br>=WARNING REPORT==== 24-Aug-2012::10:29:12 ===<br>msg_store_persistent: recovery terms differ from present<br>
rebuilding indices from scratch<div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>Adding mirror of queue 'unit_test' in vhost '/' on node rabbit@play2: <0.261.0><div class="im">
<br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>
Adding mirror of queue 'test' in vhost '/' on node rabbit@play2: <0.263.0><div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>Adding mirror of queue 'XXXX' in vhost '/' on node rabbit@play2: <0.266.0><div class="im">
<br>
<br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>Adding mirror of queue 'XYZZY' in vhost '/' on node rabbit@play2: <0.268.0><div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br>
</div>Adding mirror of queue 'system_test' in vhost '/' on node rabbit@play2: <0.271.0><div class="im"><br>
<br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>started TCP Listener on <a href="http://0.0.0.0:5672" target="_blank">0.0.0.0:5672</a><div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>rabbit on node rabbit@play up<div class="im">
<br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>
rabbit on node rabbit@util up<div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>Management agent started.<div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br></div>Management plugin started. Port: 55672, path: /<br>
<br>And here is rabbit@util.log<div class="im"><br>
=INFO REPORT==== 24-Aug-2012::10:29:06 ===<br></div>Mirrored-queue (queue 'charon' in vhost '/'): Slave <rabbit@util.3.262.0> saw deaths of mirrors <rabbit@play2.2.271.0> <br><div class="im"><br>
=INFO REPORT==== 24-Aug-2012::10:29:06 ===<br></div>
Mirrored-queue (queue 'charon' in vhost '/'): Promoting slave <rabbit@util.3.262.0> to master<div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:06 ===<br></div>Mirrored-queue (queue 'system_test' in vhost '/'): Slave <rabbit@util.3.264.0> saw deaths of mirrors <rabbit@play2.2.273.0> <br>
<div class="im">
<br>=INFO REPORT==== 24-Aug-2012::10:29:06 ===<br></div>Mirrored-queue (queue 'system_test' in vhost '/'): Promoting slave <rabbit@util.3.264.0> to master<div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:06 ===<br>
rabbit on node rabbit@play2 down<br><br>=INFO REPORT==== 24-Aug-2012::10:29:06 ===<br></div>Mirrored-queue (queue 'cmcmd' in vhost '/'): Slave <rabbit@util.3.258.0> saw deaths of mirrors <rabbit@play2.2.267.0> <br>
<div class="im">
<br>=INFO REPORT==== 24-Aug-2012::10:29:06 ===<br></div>Mirrored-queue (queue 'cmcmd' in vhost '/'): Promoting slave <rabbit@util.3.258.0> to master<div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:06 ===<br>
</div>Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Slave <rabbit@util.3.260.0> saw deaths of mirrors <rabbit@play2.2.269.0> <br><div class="im">
<br>=INFO REPORT==== 24-Aug-2012::10:29:06 ===<br></div>Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Promoting slave <rabbit@util.3.260.0> to master<div class="im"><br><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br>
</div>
Mirrored-queue (queue 'unit_test' in vhost '/'): Master <rabbit@util.3.2457.0> saw deaths of mirrors <rabbit@play2.3.261.0> <br><div class="im"><br>=INFO REPORT==== 24-Aug-2012::10:29:12 ===<br>
rabbit on node rabbit@play2 up<br>
<br><br></div></font></span><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote">On Fri, Aug 24, 2012 at 10:51 AM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Matt,<div><br>
<br>
On 24/08/12 18:34, Matt Pietrek wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I was able to reproduce this in exactly the same way with 2.8.6 and<br>
Erlang R15B01.<br>
</blockquote>
<br></div>
Thanks.<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Here's the log from rabbit@play.log. When play2 goes down and comes back<br>
up, the 'unit_test' queue (the only queue with messages) disappears.<br>
</blockquote>
<br></div>
Just to confirm, the queues are mirrored across *three* nodes - play, play2 and util, right?<br>
<br>
The logs for play and util from around the same time might offer us some more clues.<span><font color="#888888"><br>
<br>
Matthias.<br>
</font></span></blockquote></div><br>
</div></div></blockquote></div><br>