<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><blockquote type="cite"></blockquote><div><div>Hi all,</div><div><br></div><div>When restarting a node in a cluster that contained mirrored queues, I just experienced a mirrored queue disappearing completely from the cluster.</div><div><br></div><div>Both nodes in the cluster were both ram and disc nodes.</div><div><br></div><div>This is extremely worrisome to say the least.</div><div><br></div><div>--<br><br>=INFO REPORT==== 28-Apr-2014::14:17:07 ===<br>Synchronising queue 'affiliate_clicks' in vhost '/': 5565 messages to synchronise<br><br>=INFO REPORT==== 28-Apr-2014::14:17:07 ===<br>Synchronising queue 'affiliate_clicks' in vhost '/': all slaves already synced<br><br>=INFO REPORT==== 29-Apr-2014::22:19:06 ===<br>Mirrored-queue (queue 'affiliate_clicks' in vhost '/'): Slave <rabbit@mq04.1.274.0> saw deaths of mirrors <rabbit@mq03.1.280.0> <br><br>=INFO REPORT==== 29-Apr-2014::22:19:06 ===<br>Mirrored-queue (queue 'affiliate_clicks' in vhost '/'): Promoting slave <rabbit@mq04.1.274.0> to master<br><br></div><div>=INFO REPORT==== 29-Apr-2014::22:19:33 ===<br>rabbit on node rabbit@mq03 up<br><br><br>=INFO REPORT==== 29-Apr-2014::22:19:33 ===<br>Synchronising queue 'affiliate_clicks' in vhost '/': complete<br><br>=INFO REPORT==== 29-Apr-2014::22:19:33 ===<br>Synchronising queue 'affiliate_clicks' in vhost '/': 4696 messages to synchronise<br><br>=INFO REPORT==== 29-Apr-2014::22:19:33 ===<br>Synchronising queue 'affiliate_clicks' in vhost '/': all slaves already synced<br><br></div><div><snip> lots of connection logs, then kaboom </snip></div><div><br></div><div>=INFO REPORT==== 29-Apr-2014::22:23:48 ===<br>Mirrored-queue (queue 'affiliate_clicks' in vhost '/'): Master <rabbit@mq04.1.274.0> saw deaths of mirrors <rabbit@mq03.2.277.0> <br><br><br></div><div><br>=ERROR REPORT==== 29-Apr-2014::22:23:50 ===<br>** Generic server <0.274.0> terminating<br>** Last message in was emit_stats<br>** When Server state == {q,<br> {amqqueue,<br> {resource,<<"/">>,queue,<<"affiliate_clicks">>},<br> true,false,none,[],<0.274.0>,[],[],<br> [{vhost,<<"/">>},<br> {name,<<"affiliate_queues">>},<br> {pattern,<<"^affiliate_.*$">>},<br> {definition,<br> [{<<"ha-mode">>,<<"all">>},<br> {<<"ha-sync-mode">>,<<"automatic">>}]},<br> {priority,0}],<br> [{<2827.281.0>,<2827.280.0>}]},<br> none,false,rabbit_mirror_queue_master,<br> {state,<br> {resource,<<"/">>,queue,<<"affiliate_clicks">>},<br> <0.275.0>,<0.19739.588>,rabbit_variable_queue,<br> {vqstate,<br> {0,{[],[]}},<br> {0,{[],[]}},<br> {delta,undefined,0,undefined},<br> {0,{[],[]}},<br> {2660,<br> {[{msg_status,2363798,<br> <<117,194,172,33,185,58,225,43,141,116,31,73,<br> 152,23,146,23>>,<br> {basic_message,<br> {resource,<<"/">>,exchange,<br> <<"affiliate_clicks">>},<br> [<<"#">>],<br> {content,60,<br> {'P_basic',<<"application/json">>,undefined,<br> undefined,undefined,undefined,undefined,<br> undefined,undefined,undefined,undefined,<br> undefined,undefined,undefined,undefined},<br> <<128,0,16,97,112,112,108,105,99,97,116,105,<br> 111,110,47,106,115,111,110>>,<br> rabbit_framing_amqp_0_9_1,<br> [<<"DATA SNIPPED OUT">>]},<br> <<205,79,109,87,12,83,109,226,230,122,218,63,<br> 27,68,138,67>>,<br> false},<br> false,false,false,false,<br> </div><div><br></div><div><LOTS OF REPEATING LOG DATA></div><div><br></div><div><br></div><div> 2363799,<br> {0,nil},<br> {0,nil},<br> {qistate,<br> "/var/lib/rabbitmq/mnesia/rabbit@mq04/queues/D8CDHLZOTXCZL6MJMMYRK9EAN",<br> {{dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> []},<br> undefined,0,65536,<br> #Fun<rabbit_variable_queue.2.81334491>,<br> {0,nil}},<br> {{client_msstate,msg_store_persistent,<br> <<69,37,230,131,60,26,47,62,12,194,26,130,4,129,<br> 159,57>>,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> {state,356427,<br> "/var/lib/rabbitmq/mnesia/rabbit@mq04/msg_store_persistent"},<br> rabbit_msg_store_ets_index,<br> "/var/lib/rabbitmq/mnesia/rabbit@mq04/msg_store_persistent",<br> <0.265.0>,360524,352330,364621,368718},<br> {client_msstate,msg_store_transient,<br> <<140,110,236,52,188,182,217,136,180,245,92,51,<br> 176,116,195,10>>,<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> {state,335942,<br> "/var/lib/rabbitmq/mnesia/rabbit@mq04/msg_store_transient"},<br> rabbit_msg_store_ets_index,<br> "/var/lib/rabbitmq/mnesia/rabbit@mq04/msg_store_transient",<br> <0.260.0>,340039,331840,344136,348233}},<br> true,0,2660,0,infinity,2660,2660,0,0,0,<br> {rates,<br> {{1398,824624,347232},0},<br> {{1398,824624,347232},84},<br> 0.0,17.611352475686193,<br> {1398,824629,389132}},<br> {0,nil},<br> {0,nil},<br> {0,nil},<br> {0,nil},<br> 0,0,<br> {rates,<br> {{1398,824624,347232},6706},<br> {{1398,824624,347232},0},<br> 663.4928634941101,0.0,<br> {1398,824629,389132}}},<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}},<br> [],<br> {set,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> []}}}},<br> {[],[]},<br> undefined,undefined,undefined,undefined,<br> {state,fine,5000,#Ref<0.0.527.127396>},<br> {0,nil},<br> undefined,undefined,undefined,<br> {dict,1,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br> [[<0.18782.588>|#Ref<0.0.524.252716>]]}}},<br> undefined,undefined,undefined,running}<br>** Reason for termination == <br>** {{badmatch,{error,not_found}},<br> [{rabbit_mirror_queue_master,stop_all_slaves,2},<br> {rabbit_mirror_queue_master,delete_and_terminate,2},<br> {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6},<br> {rabbit_amqqueue_process,terminate_shutdown,2},<br> {gen_server2,terminate,3},<br> {proc_lib,wake_up,3}]}<br>** In 'terminate' callback with reason ==<br>** {{badmatch,{error,not_found}},<br> [{rabbit_amqqueue_process,i,2},<br> {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2},<br> {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2},<br> {rabbit_amqqueue_process,emit_stats,2},<br> {rabbit_amqqueue_process,handle_info,2},<br> {gen_server2,handle_msg,2},<br> {proc_lib,wake_up,3}]}<br><br><br></div><div><br></div><div><br></div><div>Here's the error in the SASL log:</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 29-Apr-2014::22:23:55 ===<br> Supervisor: {local,<br> rabbit_mirror_queue_slave_sup}<br> Context: child_terminated<br> Reason: {{badmatch,{error,not_found}},<br> [{rabbit_mirror_queue_master,stop_all_slaves,2},<br> {rabbit_mirror_queue_master,delete_and_terminate,2},<br> {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6},<br> {rabbit_amqqueue_process,terminate_shutdown,2},<br> {gen_server2,terminate,3},<br> {proc_lib,wake_up,3}]}<br> Offender: [{pid,<0.274.0>},<br> {name,rabbit_mirror_queue_slave},<br> {mfa,<br> {rabbit_mirror_queue_slave,start_link,<br> [{amqqueue,<br> {resource,<<"/">>,queue,<<"affiliate_clicks">>},<br> true,false,none,[],<2827.280.0>,[],[],<br> [{vhost,<<"/">>},<br> {name,<<"affiliate_queues">>},<br> {pattern,<<"^affiliate_.*$">>},<br> {definition,<br> [{<<"ha-mode">>,<<"all">>},<br> {<<"ha-sync-mode">>,<<"automatic">>}]},<br> {priority,0}],<br> [{<2827.281.0>,<2827.280.0>}]}]}},<br> {restart_type,temporary},<br> {shutdown,4294967295},<br> {child_type,worker}]</div></div><div><br></div><div><br></div><div>Known issue? Need to update? Please advise.</div><div><br></div><div>Cheers,</div><div><br></div><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Mark Steele, CISSP, CSM, GCIA, GPEN<br>Director of development<br>Instaclick Inc.<br><a href="mailto:marks@nationalfibre.net">marks@nationalfibre.net</a><br>m: (416) 844-9221</div>
</div>
<br></body></html>