I'll be the first to admit that this is a somewhat unusual/complicated circumstance, but I'm posting this on the chance there's something unusual. This is RabbitMQ is <b>2.8.7 </b>on Erlang R15B01.<br><br>We have two brokers (mq1, mq2) running clustered with all HA queues. A VIP is in front of them. <br>
<br>At some point there was a network event, and afterwards, a queue vanished. (Still trying to figure out exactly what the network hiccup was, but it's fair to say there was a communication disruption between the nodes.) My guess from looking at the logs is that each node had a moment where it couldn't see the other, so took actions that were ultimately incompatible when both nodes were visible to each other.<br>
<br>The actual error as seen by our code is a missing queue named "charon". You'll see it near the end of the rabbit@mq2.log (below):<br><br><b><span style="color:rgb(255,0,0)">{amqp_error,not_found,"no queue 'charon' in vhost '/'",'queue.declare'}</span></b><br>
<br>Prior to the network event, our client was successfully reading from the charon queue.<br><br><br><b>Here's the relevant parts of the rabbit@mq1.log file<br>====================================================</b><br>
=INFO REPORT==== 11-Oct-2012::14:01:58 ===<br>rabbit on node rabbit@mq2 down<br><br>=ERROR REPORT==== 11-Oct-2012::14:02:03 ===<br>Mnesia(rabbit@mq1): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@mq2}<br>
<br>=ERROR REPORT==== 11-Oct-2012::14:02:03 ===<br>** Generic server <0.547.0> terminating<br>** Last message in was {mnesia_locker,rabbit@mq2,granted}<br>** When Server state == {state,<br> {0,<0.547.0>},<br>
{{0,<0.547.0>},undefined},<br> {{0,<0.547.0>},undefined},<br> {resource,<<"/">>,queue,<<"charon">>},<br>
rabbit_mirror_queue_coordinator,<br> {3,<br> [{{0,<0.547.0>},<br> {view_member,<br> {0,<0.547.0>},<br>
[],<br> {0,<0.547.0>},<br> {0,<0.547.0>}}}]},<br> 1,[],<br> [<0.546.0>],<br>
{[],[]},<br> [],undefined}<br>** Reason for termination == <br>** {function_clause,<br> [{gm,handle_info,<br> [{mnesia_locker,rabbit@mq2,granted},<br>
{state,<br> {0,<0.547.0>},<br> {{0,<0.547.0>},undefined},<br> {{0,<0.547.0>},undefined},<br> {resource,<<"/">>,queue,<<"charon">>},<br>
rabbit_mirror_queue_coordinator,<br> {3,<br> [{{0,<0.547.0>},<br> {view_member,<br> {0,<0.547.0>},<br> [],<br>
{0,<0.547.0>},<br> {0,<0.547.0>}}}]},<br> 1,[],<br> [<0.546.0>],<br> {[],[]},<br> [],undefined}],<br>
[{file,"src/gm.erl"},{line,665}]},<br> {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,912}]},<br> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}<br>
<br>=ERROR REPORT==== 11-Oct-2012::14:02:03 ===<br>** Generic server <0.545.0> terminating<br>** Last message in was {'EXIT',<0.546.0>,<br> {function_clause,<br> [{gm,handle_info,<br>
[{mnesia_locker,rabbit@mq2,granted},<br> {state,<br> {0,<0.547.0>},<br> {{0,<0.547.0>},undefined},<br>
{{0,<0.547.0>},undefined},<br> {resource,<<"/">>,queue,<<"charon">>},<br> rabbit_mirror_queue_coordinator,<br>
{3,<br> [{{0,<0.547.0>},<br> {view_member,<br> {0,<0.547.0>},<br> [],<br>
{0,<0.547.0>},<br> {0,<0.547.0>}}}]},<br> 1,[],<br> [<0.546.0>],<br> {[],[]},<br>
[],undefined}],<br> [{file,"src/gm.erl"},{line,665}]},<br> {gen_server2,handle_msg,2,<br> [{file,"src/gen_server2.erl"},{line,912}]},<br>
{proc_lib,wake_up,3,<br> [{file,"proc_lib.erl"},{line,237}]}]}}<br>** When Server state == {q,<br> {amqqueue,<br> {resource,<<"/">>,queue,<<"charon">>},<br>
true,false,none,<br> [{<<"x-ha-policy">>,longstr,<<"all">>}],<br> <0.545.0>,[],all},<br> none,false,rabbit_mirror_queue_master,<br>
{state,<0.547.0>,<0.546.0>,rabbit_variable_queue,<br> {vqstate,<br> {0,{[],[]}},<br> {0,{[],[]}},<br> {delta,undefined,0,undefined},<br>
{0,{[],[]}},<br> {0,{[],[]}},<br> 0,<br> {0,nil},<br> undefined,<br> {0,nil},<br>
{qistate,<br> "/highland/var/lib/rabbit@mq1/queues/4L2SY4TVRPM8HP85JDVO85BPL",<br> {{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> []},<br> undefined,0,262144,<br>
#Fun<rabbit_variable_queue.2.121500735>,<br> {0,nil}},<br> {{client_msstate,msg_store_persistent,<br> <<112,32,84,92,54,73,67,43,231,35,164,79,106,8,27,<br>
132>>,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []},<br>
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> {state,237637,<br> "/highland/var/lib/rabbit@mq1/msg_store_persistent"},<br>
rabbit_msg_store_ets_index,<br> "/highland/var/lib/rabbit@mq1/msg_store_persistent",<br> <0.289.0>,241734,233540,245831,249928},<br>
{client_msstate,msg_store_transient,<br> <<204,18,155,40,29,120,124,76,144,83,195,135,250,<br> 91,216,3>>,<br> {dict,0,16,16,8,80,48,<br>
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br>
{state,217152,<br> "/highland/var/lib/rabbit@mq1/msg_store_transient"},<br> rabbit_msg_store_ets_index,<br> "/highland/var/lib/rabbit@mq1/msg_store_transient",<br>
<0.284.0>,221249,213055,225346,229443}},<br> true,0,#Fun<rabbit_amqqueue_process.5.24780734>,0,<br> 0,infinity,0,0,0,0,0,<br> {rates,<br>
{{1349,988602,967125},0},<br> {{1349,988602,967125},0},<br> 0.0,0.0,<br> {1349,988606,141019}},<br> {0,nil},<br>
{0,nil},<br> {0,nil},<br> {0,nil},<br> 0,0,<br> {rates,<br> {{1349,988602,967125},0},<br>
{{1349,988602,967125},0},<br> 0.0,0.0,<br> {1349,988606,141019}}},<br> 0,<br> {dict,0,16,16,8,80,48,<br>
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> [],<br>
{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br>
{set,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}}},<br>
{[],[]},<br> undefined,undefined,undefined,undefined,<br> {state,fine,5000,undefined},<br> {0,nil},<br> undefined,undefined,undefined,<br>
{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br> 1,<br>
{{0,nil},{0,nil}},<br> undefined,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br>
undefined,undefined}<br>** Reason for termination == <br>** {noproc,{gen_server2,call,[<0.547.0>,info,infinity]}}<br>** In 'terminate' callback with reason ==<br>** {function_clause,<br>
[{gm,handle_info,<br> [{mnesia_locker,rabbit@mq2,granted},<br> {state,<br> {0,<0.547.0>},<br> {{0,<0.547.0>},undefined},<br> {{0,<0.547.0>},undefined},<br>
{resource,<<"/">>,queue,<<"charon">>},<br> rabbit_mirror_queue_coordinator,<br> {3,<br> [{{0,<0.547.0>},<br>
{view_member,<br> {0,<0.547.0>},<br> [],<br> {0,<0.547.0>},<br> {0,<0.547.0>}}}]},<br> 1,[],<br>
[<0.546.0>],<br> {[],[]},<br> [],undefined}],<br> [{file,"src/gm.erl"},{line,665}]},<br> {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,912}]},<br>
{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}<br><br>=INFO REPORT==== 11-Oct-2012::14:02:03 ===<br>Mirrored-queue (queue 'cmcmd' in vhost '/'): Master <rabbit@mq1.2.665.0> saw deaths of mirrors <rabbit@mq2.1.383.0> <br>
<br>=INFO REPORT==== 11-Oct-2012::14:02:03 ===<br>Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Master <rabbit@mq1.2.605.0> saw deaths of mirrors <rabbit@mq2.1.377.0> <br><br>=INFO REPORT==== 11-Oct-2012::14:02:03 ===<br>
global: Name conflict terminating {rabbit_mgmt_db,<3198.32532.0>}<br><br>=INFO REPORT==== 11-Oct-2012::14:26:40 ===<br>accepting AMQP connection <0.6507.1> (<a href="http://10.72.0.50:46742">10.72.0.50:46742</a> -> <a href="http://10.72.0.47:5672">10.72.0.47:5672</a>)<br>
<br>=INFO REPORT==== 11-Oct-2012::14:26:40 ===<br>closing AMQP connection <0.6507.1> (<a href="http://10.72.0.50:46742">10.72.0.50:46742</a> -> <a href="http://10.72.0.47:5672">10.72.0.47:5672</a>)<br><br><br><br>
<br><b>And here's the relevant parts of the rabbit@m2.log file<br>====================================================</b><br>=ERROR REPORT==== 11-Oct-2012::14:01:37 ===<br>** Node rabbit@mq1 not responding **<br>** Removing (timedout) connection **<br>
<br>=INFO REPORT==== 11-Oct-2012::14:01:37 ===<br>rabbit on node rabbit@mq1 down<br><br>=INFO REPORT==== 11-Oct-2012::14:01:40 ===<br>Statistics database started.<br><br>=INFO REPORT==== 11-Oct-2012::14:01:43 ===<br>Mirrored-queue (queue 'cmcmd' in vhost '/'): Slave <rabbit@mq2.1.383.0> saw deaths of mirrors <rabbit@mq1.2.665.0> <br>
<br>=INFO REPORT==== 11-Oct-2012::14:01:43 ===<br>Mirrored-queue (queue 'cmcmd' in vhost '/'): Promoting slave <rabbit@mq2.1.383.0> to master<br><br>=INFO REPORT==== 11-Oct-2012::14:01:43 ===<br>Mirrored-queue (queue 'charon' in vhost '/'): Slave <rabbit@mq2.1.371.0> saw deaths of mirrors <rabbit@mq1.2.545.0> <br>
<br>=INFO REPORT==== 11-Oct-2012::14:01:43 ===<br>Mirrored-queue (queue 'charon' in vhost '/'): Promoting slave <rabbit@mq2.1.371.0> to master<br><br>=INFO REPORT==== 11-Oct-2012::14:01:43 ===<br>Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Slave <rabbit@mq2.1.377.0> saw deaths of mirrors <rabbit@mq1.2.605.0> <br>
<br>=INFO REPORT==== 11-Oct-2012::14:01:43 ===<br>Mirrored-queue (queue 'ConfigurationManager' in vhost '/'): Promoting slave <rabbit@mq2.1.377.0> to master<br><br>=ERROR REPORT==== 11-Oct-2012::14:05:03 ===<br>
Mnesia(rabbit@mq2): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@mq1}<br><br>=INFO REPORT==== 11-Oct-2012::14:05:03 ===<br>Statistics database started.<br><br>=ERROR REPORT==== 11-Oct-2012::14:29:39 ===<br>
** Generic server <0.32583.0> terminating<br>** Last message in was {'$gen_cast',{gm_deaths,[<0.5504.1>]}}<br>** When Server state == {state,<br> {amqqueue,<br> {resource,<<"/">>,queue,<<"charon">>},<br>
true,false,none,<br> [{<<"x-ha-policy">>,longstr,<<"all">>}],<br> <0.371.0>,[],all},<br>
<0.372.0>,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []},<br>
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> [],[]}}},<br> #Fun<rabbit_mirror_queue_master.1.32244930>,<br> #Fun<rabbit_mirror_queue_master.2.32244930>}<br>
** Reason for termination == <br>** {bad_return_value,<br> {error,<br> {{badmatch,[]},<br> [{rabbit_mirror_queue_misc,'-remove_from_queue/2-fun-0-',2,<br> [{file,"src/rabbit_mirror_queue_misc.erl"},{line,63}]},<br>
{mnesia_tm,apply_fun,3,[{file,"mnesia_tm.erl"},{line,829}]},<br> {mnesia_tm,execute_transaction,5,<br> [{file,"mnesia_tm.erl"},{line,809}]},<br> {rabbit_misc,'-execute_mnesia_transaction/1-fun-0-',1,<br>
[{file,"src/rabbit_misc.erl"},{line,440}]},<br> {worker_pool_worker,handle_call,3,<br> [{file,"src/worker_pool_worker.erl"},{line,80}]},<br> {gen_server2,handle_msg,2,<br>
[{file,"src/gen_server2.erl"},{line,879}]},<br> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}}}<br><br>=ERROR REPORT==== 11-Oct-2012::14:29:39 ===<br>** Generic server <0.371.0> terminating<br>
** Last message in was {'EXIT',<0.32583.0>,<br> {bad_return_value,<br> {error,<br> {{badmatch,[]},<br> [{rabbit_mirror_queue_misc,<br>
'-remove_from_queue/2-fun-0-',2,<br> [{file,"src/rabbit_mirror_queue_misc.erl"},<br> {line,63}]},<br> {mnesia_tm,apply_fun,3,<br>
[{file,"mnesia_tm.erl"},{line,829}]},<br> {mnesia_tm,execute_transaction,5,<br> [{file,"mnesia_tm.erl"},{line,809}]},<br>
{rabbit_misc,<br> '-execute_mnesia_transaction/1-fun-0-',1,<br> [{file,"src/rabbit_misc.erl"},{line,440}]},<br> {worker_pool_worker,handle_call,3,<br>
[{file,"src/worker_pool_worker.erl"},{line,80}]},<br> {gen_server2,handle_msg,2,<br> [{file,"src/gen_server2.erl"},{line,879}]},<br>
{proc_lib,wake_up,3,<br> [{file,"proc_lib.erl"},{line,237}]}]}}}}<br>** When Server state == {q,<br> {amqqueue,<br> {resource,<<"/">>,queue,<<"charon">>},<br>
true,false,none,<br> [{<<"x-ha-policy">>,longstr,<<"all">>}],<br> <0.371.0>,[],all},<br> none,false,rabbit_mirror_queue_master,<br>
{state,<0.372.0>,<0.32583.0>,rabbit_variable_queue,<br> {vqstate,<br> {0,{[],[]}},<br> {0,{[],[]}},<br> {delta,undefined,0,undefined},<br>
{0,{[],[]}},<br> {0,{[],[]}},<br> 0,<br> {0,nil},<br> undefined,<br> {0,nil},<br>
{qistate,<br> "/highland/var/lib/rabbit@mq2/queues/4L2SY4TVRPM8HP85JDVO85BPL",<br> {{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> []},<br> undefined,0,262144,<br>
#Fun<rabbit_variable_queue.2.121500735>,<br> {0,nil}},<br> {{client_msstate,msg_store_persistent,<br> <<23,245,132,154,3,96,225,184,108,75,232,224,184,<br>
114,108,141>>,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []},<br>
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> {state,282693,<br> "/highland/var/lib/rabbit@mq2/msg_store_persistent"},<br>
rabbit_msg_store_ets_index,<br> "/highland/var/lib/rabbit@mq2/msg_store_persistent",<br> <0.285.0>,286790,278596,290887,294984},<br>
{client_msstate,msg_store_transient,<br> <<17,62,75,125,126,21,66,37,170,44,198,72,18,94,<br> 170,197>>,<br> {dict,0,16,16,8,80,48,<br>
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br>
{state,262208,<br> "/highland/var/lib/rabbit@mq2/msg_store_transient"},<br> rabbit_msg_store_ets_index,<br> "/highland/var/lib/rabbit@mq2/msg_store_transient",<br>
<0.280.0>,266305,258111,270402,274499}},<br> true,0,#Fun<rabbit_mirror_queue_slave.1.69615989>,<br> 0,0,infinity,0,0,0,0,0,<br> {rates,<br>
{{1349,989325,677031},0},<br> {{1349,989325,677031},0},<br> 0.0,0.0,<br> {1349,989505,348090}},<br> {0,nil},<br>
{0,nil},<br> {0,nil},<br> {0,nil},<br> 0,0,<br> {rates,<br> {{1349,989325,677031},0},<br>
{{1349,989325,677031},0},<br> 0.0,0.0,<br> {1349,989505,348090}}},<br> 0,<br> {dict,0,16,16,8,80,48,<br>
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> [],<br>
{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br>
{set,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}}},<br>
{[],[]},<br> undefined,undefined,undefined,undefined,<br> {state,fine,5000,undefined},<br> {0,nil},<br> undefined,undefined,undefined,<br>
{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br> 1,<br>
{{0,nil},{0,nil}},<br> undefined,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br>
undefined,undefined}<br>** Reason for termination == <br>** {noproc,{gen_server2,call,[<0.372.0>,info,infinity]}}<br>** In 'terminate' callback with reason ==<br>** {bad_return_value,<br>
{error,<br> {{badmatch,[]},<br> [{rabbit_mirror_queue_misc,'-remove_from_queue/2-fun-0-',2,<br> [{file,"src/rabbit_mirror_queue_misc.erl"},{line,63}]},<br> {mnesia_tm,apply_fun,3,[{file,"mnesia_tm.erl"},{line,829}]},<br>
{mnesia_tm,execute_transaction,5,<br> [{file,"mnesia_tm.erl"},{line,809}]},<br> {rabbit_misc,'-execute_mnesia_transaction/1-fun-0-',1,<br> [{file,"src/rabbit_misc.erl"},{line,440}]},<br>
{worker_pool_worker,handle_call,3,<br> [{file,"src/worker_pool_worker.erl"},{line,80}]},<br> {gen_server2,handle_msg,2,<br> [{file,"src/gen_server2.erl"},{line,879}]},<br>
{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}}}<br><br>=INFO REPORT==== 11-Oct-2012::14:29:43 ===<br>accepting AMQP connection <0.5524.1> (<a href="http://10.72.0.23:36056">10.72.0.23:36056</a> -> <a href="http://10.72.0.56:5672">10.72.0.56:5672</a>)<br>
<br>=INFO REPORT==== 11-Oct-2012::14:29:44 ===<br>accepting AMQP connection <0.5574.1> (<a href="http://10.72.0.23:36064">10.72.0.23:36064</a> -> <a href="http://10.72.0.56:5672">10.72.0.56:5672</a>)<br><br>=INFO REPORT==== 11-Oct-2012::17:56:16 ===<br>
accepting AMQP connection <0.23868.1> (<a href="http://10.72.0.53:57122">10.72.0.53:57122</a> -> <a href="http://10.72.0.56:5672">10.72.0.56:5672</a>)<br><br>=ERROR REPORT==== 11-Oct-2012::17:56:16 ===<br>connection <0.23868.1>, channel 1 - error:<br>
{amqp_error,not_found,"no queue 'charon' in vhost '/'",'queue.declare'}<br><br>=WARNING REPORT==== 11-Oct-2012::17:56:16 ===<br>closing AMQP connection <0.23868.1> (<a href="http://10.72.0.53:57122">10.72.0.53:57122</a> -> <a href="http://10.72.0.56:5672">10.72.0.56:5672</a>):<br>
connection_closed_abruptly<br><br>=INFO REPORT==== 11-Oct-2012::17:56:16 ===<br>accepting AMQP connection <0.23878.1> (<a href="http://10.72.0.52:56025">10.72.0.52:56025</a> -> <a href="http://10.72.0.56:5672">10.72.0.56:5672</a>)<br>
<br>=ERROR REPORT==== 11-Oct-2012::17:56:16 ===<br>connection <0.23878.1>, channel 1 - error:<br>{amqp_error,not_found,"no queue 'charon' in vhost '/'",'queue.declare'}<br><br>