I'm experiencing a consistently reproducible scenario where an HA queue disappears when a node rejoins the cluster. This is using RabbitMQ 2.8.5 and Erlang R15B01.<br><br>The brokers here are play, play2 and util. I have a set of queues mirrored on all the brokers, including a durable queue called 'unit_test'.<br>
<br>Using the management UI, I publish some dummy messages to unit_test and verify that they show up in the queue, and that the queue is synched on all nodes.<br><br>I then take down the play node and start it back up. Afterwards, I force everything to be synchronized by doing a management API 'get messages" with requeue=True. When this completes, everything shows up synched as expected.<br>
<br>I then bring down play2 and start it back up. Immediately after it starts, the unit_test disappears. Interestingly, none of the other (empty) queues disappear - Only the unit_test queue.<br><br>I have the following from rabbit@play.log file:<br>
<br>=ERROR REPORT==== 22-Aug-2012::14:42:42 ===<br>** Generic server <0.278.0> terminating<br>** Last message in was {'$gen_cast',{gm,{length,4}}}<br>** When Server state == {state,<br> {amqqueue,<br>
{resource,<<"/">>,queue,<<"unit_test">>},<br> true,false,none,<br> [{<<"x-ha-policy">>,longstr,<<"all">>}],<br>
<3299.4430.0>,<br> [<3298.2580.0>],<br> all},<br> <0.280.0>,<3299.4430.0>,rabbit_variable_queue,<br>
{vqstate,<br> {0,{[],[]}},<br> {0,{[],[]}},<br> {delta,undefined,0,undefined},<br> {0,{[],[]}},<br>
{0,{[],[]}},<br> 0,<br> {0,nil},<br> undefined,<br> {0,nil},<br> {qistate,<br>
"/home/mpietrek/work/var/lib/rabbit@play/queues/5CPQWZ6GS6L07B10CDNMTM42C",<br> {{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br>
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> []},<br> undefined,0,262144,<br> #Fun<rabbit_variable_queue.2.121033067>,<br>
{0,nil}},<br> {{client_msstate,msg_store_persistent,<br> <<3,210,105,14,156,98,212,28,20,242,100,176,253,<br> 212,5,205>>,<br>
{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br>
{state,327753,<br> "/home/mpietrek/work/var/lib/rabbit@play/msg_store_persistent"},<br> rabbit_msg_store_ets_index,<br> "/home/mpietrek/work/var/lib/rabbit@play/msg_store_persistent",<br>
<0.264.0>,331850,323656,335947,340044},<br> {client_msstate,msg_store_transient,<br> <<178,59,10,119,58,171,186,219,211,141,219,166,46,<br>
89,222,107>>,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
[]}}},<br> {state,307268,<br> "/home/mpietrek/work/var/lib/rabbit@play/msg_store_transient"},<br> rabbit_msg_store_ets_index,<br>
"/home/mpietrek/work/var/lib/rabbit@play/msg_store_transient",<br> <0.259.0>,311365,303166,315462,319559}},<br> true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,<br>
0,infinity,0,0,0,0,0,<br> {rates,<br> {{1345,671562,425012},0},<br> {{1345,671562,425012},0},<br> 0.0,0.0,<br>
{1345,671606,221014}},<br> {0,nil},<br> {0,nil},<br> {0,nil},<br> {0,nil},<br> 0,0,<br>
{rates,<br> {{1345,671562,425012},0},<br> {{1345,671562,425012},0},<br> 0.0,0.0,<br> {1345,671606,221014}}},<br>
undefined,#Ref<0.0.0.8008>,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br>
{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br> 0,<br>
{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br> {dict,0,16,16,8,80,48,<br>
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br> true}<br>** Reason for termination == <br>
** {function_clause,<br> [{rabbit_mirror_queue_slave,set_synchronised,<br> [false,<br> {state,<br> {amqqueue,<br> {resource,<<"/">>,queue,<<"unit_test">>},<br>
true,false,none,<br> [{<<"x-ha-policy">>,longstr,<<"all">>}],<br> <3299.4430.0>,<br> [<3298.2580.0>],<br>
all},<br> <0.280.0>,<3299.4430.0>,rabbit_variable_queue,<br> {vqstate,<br> {0,{[],[]}},<br> {0,{[],[]}},<br> {delta,undefined,0,undefined},<br>
{0,{[],[]}},<br> {0,{[],[]}},<br> 0,<br> {0,nil},<br> undefined,<br> {0,nil},<br> {qistate,<br>
"/home/mpietrek/work/var/lib/rabbit@play/queues/5CPQWZ6GS6L07B10CDNMTM42C",<br> {{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> []},<br> undefined,0,262144,<br>
#Fun<rabbit_variable_queue.2.121033067>,<br> {0,nil}},<br> {{client_msstate,msg_store_persistent,<br> <<3,210,105,14,156,98,212,28,20,242,100,176,253,212,<br>
5,205>>,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
[]}}},<br> {state,327753,<br> "/home/mpietrek/work/var/lib/rabbit@play/msg_store_persistent"},<br> rabbit_msg_store_ets_index,<br>
"/home/mpietrek/work/var/lib/rabbit@play/msg_store_persistent",<br> <0.264.0>,331850,323656,335947,340044},<br> {client_msstate,msg_store_transient,<br>
<<178,59,10,119,58,171,186,219,211,141,219,166,46,89,<br> 222,107>>,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> {state,307268,<br> "/home/mpietrek/work/var/lib/rabbit@play/msg_store_transient"},<br>
rabbit_msg_store_ets_index,<br> "/home/mpietrek/work/var/lib/rabbit@play/msg_store_transient",<br> <0.259.0>,311365,303166,315462,319559}},<br>
true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,0,<br> infinity,0,0,0,0,0,<br> {rates,<br> {{1345,671562,425012},0},<br> {{1345,671562,425012},0},<br>
0.0,0.0,<br> {1345,671606,221014}},<br> {0,nil},<br> {0,nil},<br> {0,nil},<br> {0,nil},<br> 0,0,<br>
{rates,<br> {{1345,671562,425012},0},<br> {{1345,671562,425012},0},<br> 0.0,0.0,<br> {1345,671606,221014}}},<br>
undefined,#Ref<0.0.0.8008>,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br>
{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br> 0,<br> {dict,0,16,16,8,80,48,<br>
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br>
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br> true}],<br> []},<br> {rabbit_mirror_queue_slave,process_instruction,2,[]},<br> {rabbit_mirror_queue_slave,handle_cast,2,[]},<br>
{gen_server2,handle_msg,2,[]},<br> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}<br><br>=INFO REPORT==== 22-Aug-2012::14:42:42 ===<br>rabbit on node rabbit@play2 up<br>--------<br>On play2, the console spew includes this:<br>
<br>Mnesia(rabbit@play2): write performed by {tid,3817,<3297.298.0>} on record:<br> {gm_group,{resource,<<"/">>,queue,<<"system_test">>},<br> 83,<br>
[{70,<3299.289.0>},{79,<3297.298.0>},{82,<0.295.0>}]}<br>Mnesia(rabbit@play2): write performed by {tid,3817,<0.277.0>} on record:<br> {gm_group,{resource,<<"/">>,queue,<<"unit_test">>},<br>
10,<br> [{12,<3299.4432.0>},{7,<0.277.0>}]}<br>done<br>-- message delivery logic ready<br>starting error log relay ...Mnesia(rabbit@play2): Restarting transaction {tid,3818,<0.165.0>}: in 8ms {cyclic,rabbit@play2,{rabbit_queue,{resource,<<47>>,queue,<<117,110,105,116,95,116,101,115,116>>}},read,read,{tid,3816,<3297.184.0>}}<br>
done<br>starting networking ...Mnesia(rabbit@play2): delete_object performed by {tid,3816,<3297.184.0>} on record:<br> {rabbit_reverse_route,<br> {reverse_binding,<br>
{resource,<<"/">>,queue,<<"unit_test">>},<br> <<"unit_test">>,<br> {resource,<<"/">>,exchange,<<>>},<br>
[]},<br> const}<br>Mnesia(rabbit@play2): delete_object performed by {tid,3816,<3297.184.0>} on record:<br> {rabbit_route,{binding,{resource,<<"/">>,exchange,<<>>},<br>
<<"unit_test">>,<br> {resource,<<"/">>,queue,<<"unit_test">>},<br> []},<br>
const}<br>Mnesia(rabbit@play2): delete performed by {tid,3816,<3297.184.0>} on record:<br> {rabbit_queue,{resource,<<"/">>,queue,<<"unit_test">>}}<br>
Mnesia(rabbit@play2): delete_object performed by {tid,3816,<3297.184.0>} on record:<br> {rabbit_semi_durable_route,<br> {binding,<br> {resource,<<"/">>,exchange,<<>>},<br>
<<"unit_test">>,<br> {resource,<<"/">>,queue,<<"unit_test">>},<br> []},<br> const}<br>Mnesia(rabbit@play2): delete_object performed by {tid,3816,<3297.184.0>} on record:<br>
{rabbit_durable_route,<br> {binding,<br> {resource,<<"/">>,exchange,<<>>},<br> <<"unit_test">>,<br> {resource,<<"/">>,queue,<<"unit_test">>},<br>
[]},<br> const}<br>Mnesia(rabbit@play2): delete performed by {tid,3816,<3297.184.0>} on record:<br> {rabbit_durable_queue,{resource,<<"/">>,queue,<<"unit_test">>}}<br>
Mnesia(rabbit@play2): write performed by {dirty,<0.301.0>} on record:<br> {rabbit_listener,rabbit@play2,amqp,<br> "<a href="http://play2.mpietrek.internal.illumita.com">play2.mpietrek.internal.illumita.com</a>",<br>
{0,0,0,0},<br> 5672}<br>done<br>starting direct client ...done<br>starting notify cluster nodes ...done<br>
<br>broker running<br>Mnesia(rabbit@play2): write performed by {tid,3820,<3299.4432.0>} on record:<br> {gm_group,{resource,<<"/">>,queue,<<"unit_test">>},<br> 11,<br>
[{12,<3299.4432.0>},{dead,{7,<0.277.0>}}]}<br>Mnesia(rabbit@play2): write performed by {tid,3821,<3299.4432.0>} on record:<br> {gm_group,{resource,<<"/">>,queue,<<"unit_test">>},<br>
12,<br> [{12,<3299.4432.0>}]}<br>Mnesia(rabbit@play2): Transaction log dump initiated by time_threshold: {needs_dump,47}<br>Mnesia(rabbit@play2): Transaction log dump initiated by time_threshold: already_dumped<br>
<br>