[rabbitmq-discuss] HA queue disappears when a node rejoins the cluster

[Tim Watson]

Hi Matt,

Matthias has identified the cause of this and a bug has been raised to fix it. I presume this will go into a bugfix release asap, but don't take that as gospel as that decision hasn't been made yet. We certainly haven't forgot about this, that's for sure! :)

Cheers,
Tim

On 28 Aug 2012, at 19:12, Matt Pietrek wrote:

> Just pinging this thread as it seems to have gone dead, and it feels like a serious issue with clustering/HA in 2.8.6.
> 
> When we last left off, Matthias had asked for the logs for the two other brokers when the queue disappeared. My last message contained the contents of those logs.
> 
> Thanks,
> 
> Matt
> 
> On Fri, Aug 24, 2012 at 11:22 AM, Matt Pietrek <mpietrek at skytap.com> wrote:
> > The logs for play and util from around the same time might offer us some more clues.
> 
> 
> What I sent was rabbit at play.log.
> 
> rabbit at play2.log doesn't look nearly as interesting:
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Limiting to approx 924 file handles (829 sockets)
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Memory limit set to 399MB of 999MB total.
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Disk free limit set to 100MB
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Management plugin upgraded statistics to fine.
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> msg_store_transient: using rabbit_msg_store_ets_index to provide index
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> msg_store_persistent: using rabbit_msg_store_ets_index to provide index
> 
> =WARNING REPORT==== 24-Aug-2012::10:29:12 ===
> msg_store_persistent: recovery terms differ from present
> rebuilding indices from scratch
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Adding mirror of queue 'unit_test' in vhost '/' on node rabbit at play2: <0.261.0>
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Adding mirror of queue 'test' in vhost '/' on node rabbit at play2: <0.263.0>
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Adding mirror of queue 'XXXX' in vhost '/' on node rabbit at play2: <0.266.0>
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Adding mirror of queue 'XYZZY' in vhost '/' on node rabbit at play2: <0.268.0>
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Adding mirror of queue 'system_test' in vhost '/' on node rabbit at play2: <0.271.0>
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> started TCP Listener on 0.0.0.0:5672
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> rabbit on node rabbit at play up
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> rabbit on node rabbit at util up
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Management agent started.
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Management plugin started. Port: 55672, path: /
> 
> And here is rabbit at util.log
> 
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'charon' in vhost '/'): Slave <rabbit at util.3.262.0> saw deaths of mirrors <rabbit at play2.2.271.0> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'charon' in vhost '/'): Promoting slave <rabbit at util.3.262.0> to master
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'system_test' in vhost '/'): Slave <rabbit at util.3.264.0> saw deaths of mirrors <rabbit at play2.2.273.0> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'system_test' in vhost '/'): Promoting slave <rabbit at util.3.264.0> to master
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> rabbit on node rabbit at play2 down
> 
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'cmcmd' in vhost '/'): Slave <rabbit at util.3.258.0> saw deaths of mirrors <rabbit at play2.2.267.0> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'cmcmd' in vhost '/'): Promoting slave <rabbit at util.3.258.0> to master
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Slave <rabbit at util.3.260.0> saw deaths of mirrors <rabbit at play2.2.269.0> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Promoting slave <rabbit at util.3.260.0> to master
> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Mirrored-queue (queue 'unit_test' in vhost '/'): Master <rabbit at util.3.2457.0> saw deaths of mirrors <rabbit at play2.3.261.0> 
> 
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> rabbit on node rabbit at play2 up
> 
> 
> 
> On Fri, Aug 24, 2012 at 10:51 AM, Matthias Radestock <matthias at rabbitmq.com> wrote:
> Matt,
> 
> 
> On 24/08/12 18:34, Matt Pietrek wrote:
> I was able to reproduce this in exactly the same way with 2.8.6 and
> Erlang R15B01.
> 
> Thanks.
> 
> 
> Here's the log from rabbit at play.log. When play2 goes down and comes back
> up, the 'unit_test' queue (the only queue with messages) disappears.
> 
> Just to confirm, the queues are mirrored across *three* nodes - play, play2 and util, right?
> 
> The logs for play and util from around the same time might offer us some more clues.
> 
> Matthias.
> 
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120828/9129c3bf/attachment.htm>