[rabbitmq-discuss] HA queue disappears when a node rejoins the cluster

Tue Aug 28 19:59:40 BST 2012

Thanks Tim! I realize that all issues take time to fix, and certainly don't
expect instant turnaround.

In the absence of a reply, I just wanted to make sure the issue was driven
to some sort of resolution ("user error", "bug", "stray photon", etc...).

Matt

On Tue, Aug 28, 2012 at 11:24 AM, Tim Watson <tim at rabbitmq.com> wrote:

> Hi Matt,
>
> Matthias has identified the cause of this and a bug has been raised to fix
> it. I presume this will go into a bugfix release asap, but don't take that
> as gospel as that decision hasn't been made yet. We certainly haven't
> forgot about this, that's for sure! :)
>
> Cheers,
> Tim
>
> On 28 Aug 2012, at 19:12, Matt Pietrek wrote:
>
> Just pinging this thread as it seems to have gone dead, and it feels like
> a serious issue with clustering/HA in 2.8.6.
>
> When we last left off, Matthias had asked for the logs for the two other
> brokers when the queue disappeared. My last message contained the contents
> of those logs.
>
> Thanks,
>
> Matt
>
> On Fri, Aug 24, 2012 at 11:22 AM, Matt Pietrek <mpietrek at skytap.com>wrote:
>
>> > The logs for play and util from around the same time might offer us
>> some more clues.
>>
>>
>> What I sent was rabbit at play.log.
>>
>> rabbit at play2.log doesn't look nearly as interesting:
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Limiting to approx 924 file handles (829 sockets)
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Memory limit set to 399MB of 999MB total.
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Disk free limit set to 100MB
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Management plugin upgraded statistics to fine.
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> msg_store_transient: using rabbit_msg_store_ets_index to provide index
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> msg_store_persistent: using rabbit_msg_store_ets_index to provide index
>>
>> =WARNING REPORT==== 24-Aug-2012::10:29:12 ===
>> msg_store_persistent: recovery terms differ from present
>> rebuilding indices from scratch
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Adding mirror of queue 'unit_test' in vhost '/' on node rabbit at play2:
>> <0.261.0>
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Adding mirror of queue 'test' in vhost '/' on node rabbit at play2:
>> <0.263.0>
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Adding mirror of queue 'XXXX' in vhost '/' on node rabbit at play2:
>> <0.266.0>
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Adding mirror of queue 'XYZZY' in vhost '/' on node rabbit at play2:
>> <0.268.0>
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Adding mirror of queue 'system_test' in vhost '/' on node rabbit at play2:
>> <0.271.0>
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> started TCP Listener on 0.0.0.0:5672
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> rabbit on node rabbit at play up
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> rabbit on node rabbit at util up
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Management agent started.
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> Management plugin started. Port: 55672, path: /
>>
>> And here is rabbit at util.log
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
>> Mirrored-queue (queue 'charon' in vhost '/'): Slave <rabbit at util.3.262.0>
>> saw deaths of mirrors <rabbit at play2.2.271.0>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
>> Mirrored-queue (queue 'charon' in vhost '/'): Promoting slave
>> <rabbit at util.3.262.0> to master
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
>> Mirrored-queue (queue 'system_test' in vhost '/'): Slave
>> <rabbit at util.3.264.0> saw deaths of mirrors <rabbit at play2.2.273.0>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
>> Mirrored-queue (queue 'system_test' in vhost '/'): Promoting slave
>> <rabbit at util.3.264.0> to master
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
>> rabbit on node rabbit at play2 down
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
>> Mirrored-queue (queue 'cmcmd' in vhost '/'): Slave <rabbit at util.3.258.0>
>> saw deaths of mirrors <rabbit at play2.2.267.0>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
>> Mirrored-queue (queue 'cmcmd' in vhost '/'): Promoting slave
>> <rabbit at util.3.258.0> to master
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
>> Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Slave
>> <rabbit at util.3.260.0> saw deaths of mirrors <rabbit at play2.2.269.0>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:06 ===
>> Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Promoting
>> slave <rabbit at util.3.260.0> to master
>>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>>  Mirrored-queue (queue 'unit_test' in vhost '/'): Master
>> <rabbit at util.3.2457.0> saw deaths of mirrors <rabbit at play2.3.261.0>
>>
>> =INFO REPORT==== 24-Aug-2012::10:29:12 ===
>> rabbit on node rabbit at play2 up
>>
>>
>>
>> On Fri, Aug 24, 2012 at 10:51 AM, Matthias Radestock <
>> matthias at rabbitmq.com> wrote:
>>
>>> Matt,
>>>
>>>
>>> On 24/08/12 18:34, Matt Pietrek wrote:
>>>
>>>> I was able to reproduce this in exactly the same way with 2.8.6 and
>>>> Erlang R15B01.
>>>>
>>>
>>> Thanks.
>>>
>>>
>>>  Here's the log from rabbit at play.log. When play2 goes down and comes
>>>> back
>>>> up, the 'unit_test' queue (the only queue with messages) disappears.
>>>>
>>>
>>> Just to confirm, the queues are mirrored across *three* nodes - play,
>>> play2 and util, right?
>>>
>>> The logs for play and util from around the same time might offer us some
>>> more clues.
>>>
>>> Matthias.
>>>
>>
>>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120828/9354feee/attachment.htm>