[rabbitmq-discuss] HA queue disappears when a node rejoins the cluster
Matt Pietrek
mpietrek at skytap.com
Fri Aug 24 18:34:41 BST 2012
I was able to reproduce this in exactly the same way with 2.8.6 and Erlang
R15B01.
Here's the log from rabbit at play.log. When play2 goes down and comes back
up, the 'unit_test' queue (the only queue with messages) disappears.
=INFO REPORT==== 24-Aug-2012::10:29:06 ===
rabbit on node rabbit at play2 down
=INFO REPORT==== 24-Aug-2012::10:29:06 ===
Mirrored-queue (queue 'unit_test' in vhost '/'): Slave <rabbit at play.1.261.0>
saw deaths of mirrors <rabbit at play2.2.2550.0>
=INFO REPORT==== 24-Aug-2012::10:29:12 ===
rabbit on node rabbit at play2 up
=ERROR REPORT==== 24-Aug-2012::10:29:12 ===
** Generic server <0.261.0> terminating
** Last message in was {'$gen_cast',{gm,{length,3}}}
** When Server state == {state,
{amqqueue,
{resource,<<"/">>,queue,<<"unit_test">>},
true,false,none,
[{<<"x-ha-policy">>,longstr,<<"all">>}],
<3299.2457.0>,
[<3298.2550.0>],
all},
<0.262.0>,<3299.2457.0>,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
undefined,
{0,nil},
{qistate,
"/home/mpietrek/work/var/lib/rabbit at play
/queues/5CPQWZ6GS6L07B10CDNMTM42C",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
[]},
undefined,0,262144,
#Fun<rabbit_variable_queue.2.121033067>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<61,29,142,33,178,251,167,19,151,135,213,43,204,
197,156,123>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,327753,
"/home/mpietrek/work/var/lib/rabbit at play
/msg_store_persistent"},
rabbit_msg_store_ets_index,
"/home/mpietrek/work/var/lib/rabbit at play
/msg_store_persistent",
<0.255.0>,331850,323656,335947,340044},
{client_msstate,msg_store_transient,
<<41,125,255,66,12,1,141,122,87,67,196,42,155,134,
88,81>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,307268,
"/home/mpietrek/work/var/lib/rabbit at play
/msg_store_transient"},
rabbit_msg_store_ets_index,
"/home/mpietrek/work/var/lib/rabbit at play
/msg_store_transient",
<0.250.0>,311365,303144,315462,319559}},
true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,
0,infinity,0,0,0,0,0,
{rates,
{{1345,829346,423057},0},
{{1345,829346,423057},0},
0.0,0.0,
{1345,829348,736018}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0,
{rates,
{{1345,829346,423057},0},
{{1345,829346,423057},0},
0.0,0.0,
{1345,829348,736018}}},
undefined,undefined,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
0,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
true}
** Reason for termination ==
** {function_clause,
[{rabbit_mirror_queue_slave,set_synchronised,
[false,
{state,
{amqqueue,
{resource,<<"/">>,queue,<<"unit_test">>},
true,false,none,
[{<<"x-ha-policy">>,longstr,<<"all">>}],
<3299.2457.0>,
[<3298.2550.0>],
all},
<0.262.0>,<3299.2457.0>,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
undefined,
{0,nil},
{qistate,
"/home/mpietrek/work/var/lib/rabbit at play
/queues/5CPQWZ6GS6L07B10CDNMTM42C",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
[]},
undefined,0,262144,
#Fun<rabbit_variable_queue.2.121033067>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<61,29,142,33,178,251,167,19,151,135,213,43,204,197,
156,123>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,327753,
"/home/mpietrek/work/var/lib/rabbit at play
/msg_store_persistent"},
rabbit_msg_store_ets_index,
"/home/mpietrek/work/var/lib/rabbit at play
/msg_store_persistent",
<0.255.0>,331850,323656,335947,340044},
{client_msstate,msg_store_transient,
<<41,125,255,66,12,1,141,122,87,67,196,42,155,134,88,
81>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,307268,
"/home/mpietrek/work/var/lib/rabbit at play
/msg_store_transient"},
rabbit_msg_store_ets_index,
"/home/mpietrek/work/var/lib/rabbit at play
/msg_store_transient",
<0.250.0>,311365,303144,315462,319559}},
true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,0,
infinity,0,0,0,0,0,
{rates,
{{1345,829346,423057},0},
{{1345,829346,423057},0},
0.0,0.0,
{1345,829348,736018}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0,
{rates,
{{1345,829346,423057},0},
{{1345,829346,423057},0},
0.0,0.0,
{1345,829348,736018}}},
undefined,undefined,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
0,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
true}],
[]},
{rabbit_mirror_queue_slave,process_instruction,2,[]},
{rabbit_mirror_queue_slave,handle_cast,2,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}
On Wed, Aug 22, 2012 at 8:09 PM, Matthias Radestock
<matthias at rabbitmq.com>wrote:
> Matt,
>
>
> On 22/08/12 23:06, Matt Pietrek wrote:
>
>> I'm experiencing a consistently reproducible scenario where an HA queue
>> disappears when a node rejoins the cluster. This is using RabbitMQ 2.8.5
>> and Erlang R15B01.
>>
>
> please try reproducing this with 2.8.6.
>
> Matthias.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120824/682fc7a3/attachment.htm>
More information about the rabbitmq-discuss
mailing list