<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">MQ03 was the master for that queue, and it was cleanly restarted (service rabbitmq-server restart). So the chain of events was mq03 restarted, mq04 became master for that queue, mq03 rejoined after restart, then queue disappeared a few minutes later.<div><br></div><div>The sasl log on 03:</div><div><br></div><div><br>=CRASH REPORT==== 29-Apr-2014::22:24:15 ===<br> crasher:<br> initial call: gen:init_it/6<br> pid: <0.277.0><br> registered_name: []<br> exception exit: {function_clause,<br> [{gb_trees,delete_1,[4227,nil]},<br> {gb_trees,delete,2},<br> {rabbit_variable_queue,remove_pending_ack,2},<br> {rabbit_variable_queue,'-ack/2-fun-0-',2},<br> {lists,foldl,3},<br> {rabbit_variable_queue,ack,2},<br> {rabbit_mirror_queue_slave,process_instruction,2},<br> {rabbit_mirror_queue_slave,handle_cast,2}]}<br> in function gen_server2:terminate/3<br> ancestors: [rabbit_mirror_queue_slave_sup,rabbit_sup,<0.168.0>]<br> messages: [{'$gen_cast',<br> {gm,<br> {publish,<2828.25884.590>,<br> {message_properties,undefined,false},<br> {basic_message,<br> {resource,<<"/">>,exchange,<<"affiliate_clicks">>},<br> [<<"#">>],<br> {content,60,<br> {'P_basic',<<"application/json">>,undefined,undefined,<br> undefined,undefined,undefined,undefined,undefined,<br> undefined,undefined,undefined,undefined,undefined,<br> undefined},<br> <<128,0,16,97,112,112,108,105,99,97,116,105,111,110,47,<br> 106,115,111,110>>,<br> rabbit_framing_amqp_0_9_1,<br> [<<"snipped out">>]},<br> <<117,194,172,33,185,58,225,43,141,116,31,73,152,23,146,<br> 23>>,<br> false}}}},<br> {'$gen_cast',<br> {deliver,<br> {delivery,false,<2828.25884.590>,<br> {basic_message,<br> {resource,<<"/">>,exchange,<<"affiliate_clicks">>},<br> [<<"#">>],<br> {content,60,<br> {'P_basic',<<"application/json">>,undefined,undefined,<br> undefined,undefined,undefined,undefined,undefined,<br> undefined,undefined,undefined,undefined,undefined,<br> undefined},<br> <<128,0,16,97,112,112,108,105,99,97,116,105,111,110,47,<br> 106,115,111,110>>,<br> rabbit_framing_amqp_0_9_1,<br> [<<"snipped out}">>]},<br> <<117,194,172,33,185,58,225,43,141,116,31,73,152,23,146,<br> 23>>,<br> false},<br> undefined},<br> true,flow}},<br> {'$gen_cast',{gm,{sender_death,<2828.25884.590>}}},<br> {'$gen_cast',<br> {run_backing_queue,rabbit_mirror_queue_master,<br> #Fun<rabbit_mirror_queue_slave.8.82654898>}},<br> {'$gen_cast',<br> {run_backing_queue,rabbit_mirror_queue_master,<br> #Fun<rabbit_mirror_queue_slave.8.82654898>}},<br> {'EXIT',<0.278.0>,normal},<br> {'$gen_cast',<br> {run_backing_queue,rabbit_mirror_queue_master,<br> #Fun<rabbit_mirror_queue_slave.8.82654898>}},<br> {'$gen_cast',<br> {run_backing_queue,rabbit_mirror_queue_master,<br> #Fun<rabbit_mirror_queue_slave.8.82654898>}}]<br> links: [<0.273.0>]<br> dictionary: [{credit_blocked,[]},<br> {{xtype_to_module,direct},rabbit_exchange_type_direct},<br> {{xtype_to_module,topic},rabbit_exchange_type_topic},<br> {guid,{{4125214297,1894440844,1353716068,2733218191},1}}]<br> trap_exit: true<br> status: running<br> heap_size: 317811<br> stack_size: 24<br> reductions: 4292143<br> neighbours:<br><br>=SUPERVISOR REPORT==== 29-Apr-2014::22:24:15 ===<br> Supervisor: {local,<br> rabbit_mirror_queue_slave_sup}<br> Context: child_terminated<br> Reason: {function_clause,<br> [{gb_trees,delete_1,[4227,nil]},<br> {gb_trees,delete,2},<br> {rabbit_variable_queue,remove_pending_ack,2},<br> {rabbit_variable_queue,'-ack/2-fun-0-',2},<br> {lists,foldl,3},<br> {rabbit_variable_queue,ack,2},<br> {rabbit_mirror_queue_slave,process_instruction,2},<br> {rabbit_mirror_queue_slave,handle_cast,2}]}<br> Offender: [{pid,<0.277.0>},<br> {name,rabbit_mirror_queue_slave},<br> {mfa,<br> {rabbit_mirror_queue_slave,start_link,<br> [{amqqueue,<br> {resource,<<"/">>,queue,<<"affiliate_clicks">>},<br> true,false,none,[],<2828.274.0>,[],[],<br> [{vhost,<<"/">>},<br> {name,<<"affiliate_queues">>},<br> {pattern,<<"^affiliate_.*$">>},<br> {definition,<br> [{<<"ha-mode">>,<<"all">>},<br> {<<"ha-sync-mode">>,<<"automatic">>}]},<br> {priority,0}],<br> [{<2828.275.0>,<2828.274.0>}]}]}},<br> {restart_type,temporary},<br> {shutdown,4294967295},<br> {child_type,worker}]</div><div><br></div><div><br></div><div><br><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Mark Steele, CISSP, CSM, GCIA, GPEN<br>Director of development<br>Instaclick Inc.<br><a href="mailto:marks@nationalfibre.net">marks@nationalfibre.net</a><br>m: (416) 844-9221</div>
</div>
<br><div><div>On Apr 30, 2014, at 11:24 AM, Simon MacMullen <<a href="mailto:simon@rabbitmq.com">simon@rabbitmq.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">On 30/04/14 16:14, Mark Steele wrote:<br><blockquote type="cite">Known issue? Need to update? Please advise.<br></blockquote><br>There was a known issue, fixed in 3.2.1, where a crashing slave could cause the master (and other slaves of the same queue) to crash with stack traces like the ones you posted. So that's definitely a reason to upgrade.<br><br>Of course, that doesn't help us with why the first slave crashed. It's quite possibly another bug that has been fixed since, but just to be sure, could you look for and post any errors from the same time frame mentioning "affiliate_clicks" on mq03?<br><br>Cheers, Simon<br><br>-- <br>Simon MacMullen<br>RabbitMQ, Pivotal<br></blockquote></div><br></div></body></html>