[rabbitmq-discuss] HA queue disappears when a node rejoins the cluster

Matt Pietrek mpietrek at skytap.com
Fri Aug 24 18:34:41 BST 2012


I was able to reproduce this in exactly the same way with 2.8.6 and Erlang
R15B01.

Here's the log from rabbit at play.log. When play2 goes down and comes back
up, the 'unit_test' queue (the only queue with messages) disappears.

=INFO REPORT==== 24-Aug-2012::10:29:06 ===
rabbit on node rabbit at play2 down

=INFO REPORT==== 24-Aug-2012::10:29:06 ===
Mirrored-queue (queue 'unit_test' in vhost '/'): Slave <rabbit at play.1.261.0>
saw deaths of mirrors <rabbit at play2.2.2550.0>

=INFO REPORT==== 24-Aug-2012::10:29:12 ===
rabbit on node rabbit at play2 up

=ERROR REPORT==== 24-Aug-2012::10:29:12 ===
** Generic server <0.261.0> terminating
** Last message in was {'$gen_cast',{gm,{length,3}}}
** When Server state == {state,
                         {amqqueue,
                          {resource,<<"/">>,queue,<<"unit_test">>},
                          true,false,none,
                          [{<<"x-ha-policy">>,longstr,<<"all">>}],
                          <3299.2457.0>,
                          [<3298.2550.0>],
                          all},
                         <0.262.0>,<3299.2457.0>,rabbit_variable_queue,
                         {vqstate,
                          {0,{[],[]}},
                          {0,{[],[]}},
                          {delta,undefined,0,undefined},
                          {0,{[],[]}},
                          {0,{[],[]}},
                          0,
                          {0,nil},
                          undefined,
                          {0,nil},
                          {qistate,
                           "/home/mpietrek/work/var/lib/rabbit at play
/queues/5CPQWZ6GS6L07B10CDNMTM42C",
                           {{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            []},
                           undefined,0,262144,
                           #Fun<rabbit_variable_queue.2.121033067>,
                           {0,nil}},
                          {{client_msstate,msg_store_persistent,

<<61,29,142,33,178,251,167,19,151,135,213,43,204,
                              197,156,123>>,
                            {dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,327753,
                             "/home/mpietrek/work/var/lib/rabbit at play
/msg_store_persistent"},
                            rabbit_msg_store_ets_index,
                            "/home/mpietrek/work/var/lib/rabbit at play
/msg_store_persistent",
                            <0.255.0>,331850,323656,335947,340044},
                           {client_msstate,msg_store_transient,

<<41,125,255,66,12,1,141,122,87,67,196,42,155,134,
                              88,81>>,
                            {dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,307268,
                             "/home/mpietrek/work/var/lib/rabbit at play
/msg_store_transient"},
                            rabbit_msg_store_ets_index,
                            "/home/mpietrek/work/var/lib/rabbit at play
/msg_store_transient",
                            <0.250.0>,311365,303144,315462,319559}},

true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,
                          0,infinity,0,0,0,0,0,
                          {rates,
                           {{1345,829346,423057},0},
                           {{1345,829346,423057},0},
                           0.0,0.0,
                           {1345,829348,736018}},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          0,0,
                          {rates,
                           {{1345,829346,423057},0},
                           {{1345,829346,423057},0},
                           0.0,0.0,
                           {1345,829348,736018}}},
                         undefined,undefined,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         0,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         true}
** Reason for termination ==
** {function_clause,
       [{rabbit_mirror_queue_slave,set_synchronised,
            [false,
             {state,
                 {amqqueue,
                     {resource,<<"/">>,queue,<<"unit_test">>},
                     true,false,none,
                     [{<<"x-ha-policy">>,longstr,<<"all">>}],
                     <3299.2457.0>,
                     [<3298.2550.0>],
                     all},
                 <0.262.0>,<3299.2457.0>,rabbit_variable_queue,
                 {vqstate,
                     {0,{[],[]}},
                     {0,{[],[]}},
                     {delta,undefined,0,undefined},
                     {0,{[],[]}},
                     {0,{[],[]}},
                     0,
                     {0,nil},
                     undefined,
                     {0,nil},
                     {qistate,
                         "/home/mpietrek/work/var/lib/rabbit at play
/queues/5CPQWZ6GS6L07B10CDNMTM42C",
                         {{dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                          []},
                         undefined,0,262144,
                         #Fun<rabbit_variable_queue.2.121033067>,
                         {0,nil}},
                     {{client_msstate,msg_store_persistent,

<<61,29,142,33,178,251,167,19,151,135,213,43,204,197,
                            156,123>>,
                          {dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                          {state,327753,
                              "/home/mpietrek/work/var/lib/rabbit at play
/msg_store_persistent"},
                          rabbit_msg_store_ets_index,
                          "/home/mpietrek/work/var/lib/rabbit at play
/msg_store_persistent",
                          <0.255.0>,331850,323656,335947,340044},
                      {client_msstate,msg_store_transient,

<<41,125,255,66,12,1,141,122,87,67,196,42,155,134,88,
                            81>>,
                          {dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                          {state,307268,
                              "/home/mpietrek/work/var/lib/rabbit at play
/msg_store_transient"},
                          rabbit_msg_store_ets_index,
                          "/home/mpietrek/work/var/lib/rabbit at play
/msg_store_transient",
                          <0.250.0>,311365,303144,315462,319559}},
                     true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,0,
                     infinity,0,0,0,0,0,
                     {rates,
                         {{1345,829346,423057},0},
                         {{1345,829346,423057},0},
                         0.0,0.0,
                         {1345,829348,736018}},
                     {0,nil},
                     {0,nil},
                     {0,nil},
                     {0,nil},
                     0,0,
                     {rates,
                         {{1345,829346,423057},0},
                         {{1345,829346,423057},0},
                         0.0,0.0,
                         {1345,829348,736018}}},
                 undefined,undefined,
                 {dict,0,16,16,8,80,48,
                     {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                 {dict,0,16,16,8,80,48,
                     {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                 0,
                 {dict,0,16,16,8,80,48,
                     {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                 {dict,0,16,16,8,80,48,
                     {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                 true}],
            []},
        {rabbit_mirror_queue_slave,process_instruction,2,[]},
        {rabbit_mirror_queue_slave,handle_cast,2,[]},
        {gen_server2,handle_msg,2,[]},
        {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}



On Wed, Aug 22, 2012 at 8:09 PM, Matthias Radestock
<matthias at rabbitmq.com>wrote:

> Matt,
>
>
> On 22/08/12 23:06, Matt Pietrek wrote:
>
>> I'm experiencing a consistently reproducible scenario where an HA queue
>> disappears when a node rejoins the cluster. This is using RabbitMQ 2.8.5
>> and Erlang R15B01.
>>
>
> please try reproducing this with 2.8.6.
>
> Matthias.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120824/682fc7a3/attachment.htm>


More information about the rabbitmq-discuss mailing list