[rabbitmq-discuss] HA queue disappears when a node rejoins the cluster

Fri Aug 24 19:22:46 BST 2012

> The logs for play and util from around the same time might offer us some
more clues.

What I sent was rabbit at play.log.

rabbit at play2.log doesn't look nearly as interesting:

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Memory limit set to 399MB of 999MB total.

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Disk free limit set to 100MB

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Management plugin upgraded statistics to fine.

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 24-Aug-2012::10:29:12 ===
msg_store_persistent: recovery terms differ from present
rebuilding indices from scratch

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Adding mirror of queue 'unit_test' in vhost '/' on node rabbit at play2:
<0.261.0>

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Adding mirror of queue 'test' in vhost '/' on node rabbit at play2: <0.263.0>

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Adding mirror of queue 'XXXX' in vhost '/' on node rabbit at play2: <0.266.0>

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Adding mirror of queue 'XYZZY' in vhost '/' on node rabbit at play2: <0.268.0>

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Adding mirror of queue 'system_test' in vhost '/' on node rabbit at play2:
<0.271.0>

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
started TCP Listener on 0.0.0.0:5672

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
rabbit on node rabbit at play up

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
rabbit on node rabbit at util up

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Management agent started.

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Management plugin started. Port: 55672, path: /

And here is rabbit at util.log
=INFO REPORT==== 24-Aug-2012::10:29:06 ===
Mirrored-queue (queue 'charon' in vhost '/'): Slave <rabbit at util.3.262.0>
saw deaths of mirrors <rabbit at play2.2.271.0>

=INFO REPORT==== 24-Aug-2012::10:29:06 ===
Mirrored-queue (queue 'charon' in vhost '/'): Promoting slave
<rabbit at util.3.262.0> to master

=INFO REPORT==== 24-Aug-2012::10:29:06 ===
Mirrored-queue (queue 'system_test' in vhost '/'): Slave
<rabbit at util.3.264.0> saw deaths of mirrors <rabbit at play2.2.273.0>

=INFO REPORT==== 24-Aug-2012::10:29:06 ===
Mirrored-queue (queue 'system_test' in vhost '/'): Promoting slave
<rabbit at util.3.264.0> to master

=INFO REPORT==== 24-Aug-2012::10:29:06 ===
rabbit on node rabbit at play2 down

=INFO REPORT==== 24-Aug-2012::10:29:06 ===
Mirrored-queue (queue 'cmcmd' in vhost '/'): Slave <rabbit at util.3.258.0>
saw deaths of mirrors <rabbit at play2.2.267.0>

=INFO REPORT==== 24-Aug-2012::10:29:06 ===
Mirrored-queue (queue 'cmcmd' in vhost '/'): Promoting slave
<rabbit at util.3.258.0> to master

=INFO REPORT==== 24-Aug-2012::10:29:06 ===
Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Slave
<rabbit at util.3.260.0> saw deaths of mirrors <rabbit at play2.2.269.0>

=INFO REPORT==== 24-Aug-2012::10:29:06 ===
Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Promoting slave
<rabbit at util.3.260.0> to master

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
Mirrored-queue (queue 'unit_test' in vhost '/'): Master
<rabbit at util.3.2457.0> saw deaths of mirrors <rabbit at play2.3.261.0>

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
rabbit on node rabbit at play2 up

On Fri, Aug 24, 2012 at 10:51 AM, Matthias Radestock
<matthias at rabbitmq.com>wrote:

> Matt,
>
>
> On 24/08/12 18:34, Matt Pietrek wrote:
>
>> I was able to reproduce this in exactly the same way with 2.8.6 and
>> Erlang R15B01.
>>
>
> Thanks.
>
>
>  Here's the log from rabbit at play.log. When play2 goes down and comes back
>> up, the 'unit_test' queue (the only queue with messages) disappears.
>>
>
> Just to confirm, the queues are mirrored across *three* nodes - play,
> play2 and util, right?
>
> The logs for play and util from around the same time might offer us some
> more clues.
>
> Matthias.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120824/a6238aaa/attachment.htm>