[rabbitmq-discuss] HA queue disappears when a node rejoins the cluster

[Matt Pietrek]

Just pinging this thread as it seems to have gone dead, and it feels like a
serious issue with clustering/HA in 2.8.6.

When we last left off, Matthias had asked for the logs for the two other
brokers when the queue disappeared. My last message contained the contents
of those logs.

Thanks,

Matt

On Fri, Aug 24, 2012 at 11:22 AM, Matt Pietrek <mpietrek at skytap.com> wrote:

> > The logs for play and util from around the same time might offer us some
> more clues.
>
>
> What I sent was rabbit at play.log.
>
> rabbit at play2.log doesn't look nearly as interesting:
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Limiting to approx 924 file handles (829 sockets)
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Memory limit set to 399MB of 999MB total.
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Disk free limit set to 100MB
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Management plugin upgraded statistics to fine.
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> msg_store_transient: using rabbit_msg_store_ets_index to provide index
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> msg_store_persistent: using rabbit_msg_store_ets_index to provide index
>
> =WARNING REPORT==== 24-Aug-2012::10:29:12 ===
> msg_store_persistent: recovery terms differ from present
> rebuilding indices from scratch
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Adding mirror of queue 'unit_test' in vhost '/' on node rabbit at play2:
> <0.261.0>
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Adding mirror of queue 'test' in vhost '/' on node rabbit at play2: <0.263.0>
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Adding mirror of queue 'XXXX' in vhost '/' on node rabbit at play2: <0.266.0>
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Adding mirror of queue 'XYZZY' in vhost '/' on node rabbit at play2:
> <0.268.0>
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Adding mirror of queue 'system_test' in vhost '/' on node rabbit at play2:
> <0.271.0>
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> started TCP Listener on 0.0.0.0:5672
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> rabbit on node rabbit at play up
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> rabbit on node rabbit at util up
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Management agent started.
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Management plugin started. Port: 55672, path: /
>
> And here is rabbit at util.log
>
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'charon' in vhost '/'): Slave <rabbit at util.3.262.0>
> saw deaths of mirrors <rabbit at play2.2.271.0>
>
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'charon' in vhost '/'): Promoting slave
> <rabbit at util.3.262.0> to master
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'system_test' in vhost '/'): Slave
> <rabbit at util.3.264.0> saw deaths of mirrors <rabbit at play2.2.273.0>
>
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'system_test' in vhost '/'): Promoting slave
> <rabbit at util.3.264.0> to master
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> rabbit on node rabbit at play2 down
>
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'cmcmd' in vhost '/'): Slave <rabbit at util.3.258.0>
> saw deaths of mirrors <rabbit at play2.2.267.0>
>
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'cmcmd' in vhost '/'): Promoting slave
> <rabbit at util.3.258.0> to master
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Slave
> <rabbit at util.3.260.0> saw deaths of mirrors <rabbit at play2.2.269.0>
>
> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
> Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Promoting
> slave <rabbit at util.3.260.0> to master
>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> Mirrored-queue (queue 'unit_test' in vhost '/'): Master
> <rabbit at util.3.2457.0> saw deaths of mirrors <rabbit at play2.3.261.0>
>
> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
> rabbit on node rabbit at play2 up
>
>
>
> On Fri, Aug 24, 2012 at 10:51 AM, Matthias Radestock <
> matthias at rabbitmq.com> wrote:
>
>> Matt,
>>
>>
>> On 24/08/12 18:34, Matt Pietrek wrote:
>>
>>> I was able to reproduce this in exactly the same way with 2.8.6 and
>>> Erlang R15B01.
>>>
>>
>> Thanks.
>>
>>
>>  Here's the log from rabbit at play.log. When play2 goes down and comes back
>>> up, the 'unit_test' queue (the only queue with messages) disappears.
>>>
>>
>> Just to confirm, the queues are mirrored across *three* nodes - play,
>> play2 and util, right?
>>
>> The logs for play and util from around the same time might offer us some
>> more clues.
>>
>> Matthias.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120828/9febb102/attachment.htm>