Hi all, <br><br>We're running RabbitMQ/Erlang ==
2.8.1 / R14A
<br><br>We're (currently, don't get me started) using kombu to talk to RabbitMQ from some Python code. <br><br>We've come across a strange issue where, best we can tell, a consumer rejects a message, and that's followed by the complete disappearance of the queue. I'm not even sure if these two events are actually correllated (more testing is being done now to try and reliably reproduce the issue), but in checking out the queue's configuration options, I don't see a reason in the configuration that would point to something more obvious like, say, "queue has a single consumer that crashes & it's configured w/ auto-delete=true" or something.<br>
<br>Here's the only log information that seemed to be relevant. If anyone has clues or pointers, they're much appreciated. <br><br><div class="highlight"><pre><div class="line" id="LC1">=ERROR REPORT==== 3-Jul-2012::18:34:28 ===</div>
<div class="line" id="LC2">** Generic server <0.14335.436> terminating</div><div class="line" id="LC3">** Last message in was {'$gen_cast',{gm,{fold,undefined,"\b"}}}</div><div class="line" id="LC4">
** When Server state == {state,</div><div class="line" id="LC5"> {amqqueue,</div><div class="line" id="LC6"> {resource,<<"/">>,queue,</div><div class="line" id="LC7">
<<"evo.workers.do_stuff.with_things">>},</div><div class="line" id="LC8"> true,false,none,</div><div class="line" id="LC9"> [{<<"x-ha-policy">>,longstr,<<"all">>}],</div>
<div class="line" id="LC10"> <7472.27433.653>,[],all},</div><div class="line" id="LC11"> <0.14337.436>,<7472.27433.653>,rabbit_variable_queue,</div><div class="line" id="LC12">
{vqstate,</div><div class="line" id="LC13"> {0,{[],[]}},</div><div class="line" id="LC14"> {0,{[],[]}},</div><div class="line" id="LC15"> {delta,undefined,0,undefined},</div>
<div class="line" id="LC16"> {0,{[],[]}},</div><div class="line" id="LC17"> {0,{[],[]}},</div><div class="line" id="LC18"> 9,</div><div class="line" id="LC19">
{1,</div><div class="line" id="LC20"> {8,</div><div class="line" id="LC21"> {msg_status,8,</div><div class="line" id="LC22"> <<8,106,184,177,219,146,140,107,196,247,218,34,</div>
<div class="line" id="LC23"> 138,15,137,90>>,</div><div class="line" id="LC24"> undefined,true,true,true,true,</div><div class="line" id="LC25"> {message_properties,undefined,false}},</div>
<div class="line" id="LC26"> nil,nil}},</div><div class="line" id="LC27"> undefined,</div><div class="line" id="LC28"> {0,nil},</div><div class="line" id="LC29">
{qistate,</div><div class="line" id="LC30"> "/var/lib/rabbitmq/mnesia/s-rabbit1/queues/6SK5DU9WSNS31F15H65PWTGEF",</div><div class="line" id="LC31"> {{dict,0,16,16,8,80,48,</div>
<div class="line" id="LC32"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div class="line" id="LC33"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div class="line" id="LC34"> []}}},</div><div class="line" id="LC35"> [{segment,0,</div><div class="line" id="LC36"> "/var/lib/rabbitmq/mnesia/s-rabbit1/queues/6SK5DU9WSNS31F15H65PWTGEF/0.idx",</div>
<div class="line" id="LC37"> {array,16384,0,undefined,</div><div class="line" id="LC38"> {{{{{undefined,undefined,undefined,undefined,</div><div class="line" id="LC39">
undefined,undefined,undefined,undefined,</div><div class="line" id="LC40"> {{<<8,106,184,177,219,146,140,107,196,247,</div><div class="line" id="LC41">
218,34,138,15,137,90>>,</div><div class="line" id="LC42"> {message_properties,undefined,true},</div><div class="line" id="LC43"> true},</div>
<div class="line" id="LC44"> del,no_ack},</div><div class="line" id="LC45"> undefined},</div><div class="line" id="LC46"> 10,10,10,10,10,10,10,10,10,10},</div>
<div class="line" id="LC47"> 100,100,100,100,100,100,100,100,100,100},</div><div class="line" id="LC48"> 1000,1000,1000,1000,1000,1000,1000,1000,1000,</div>
<div class="line" id="LC49"> 1000},</div><div class="line" id="LC50"> 10000,10000,10000,10000,10000,10000,10000,</div><div class="line" id="LC51"> 10000,10000,10000}},</div>
<div class="line" id="LC52"> 1}]},</div><div class="line" id="LC53"> #Ref<0.0.113.112114>,2,262144,</div><div class="line" id="LC54"> #Fun<rabbit_variable_queue.2.91259159>,</div>
<div class="line" id="LC55"> {0,nil}},</div><div class="line" id="LC56"> {{client_msstate,msg_store_persistent,</div><div class="line" id="LC57"> <<45,226,10,206,192,135,238,119,205,177,43,46,24,</div>
<div class="line" id="LC58"> 159,170,52>>,</div><div class="line" id="LC59"> {dict,0,16,16,8,80,48,</div><div class="line" id="LC60"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div class="line" id="LC61"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div class="line" id="LC62"> []}}},</div><div class="line" id="LC63"> {state,311369,</div>
<div class="line" id="LC64"> "/var/lib/rabbitmq/mnesia/s-rabbit1/msg_store_persistent"},</div><div class="line" id="LC65"> rabbit_msg_store_ets_index,</div>
<div class="line" id="LC66"> "/var/lib/rabbitmq/mnesia/s-rabbit1/msg_store_persistent",</div><div class="line" id="LC67"> <0.246.0>,315466,307272,319563,323660},</div>
<div class="line" id="LC68"> {client_msstate,msg_store_transient,</div><div class="line" id="LC69"> <<172,50,239,59,161,4,42,98,116,42,171,120,108,181,</div><div class="line" id="LC70">
1,76>>,</div><div class="line" id="LC71"> {dict,0,16,16,8,80,48,</div><div class="line" id="LC72"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div class="line" id="LC73"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div class="line" id="LC74"> []}}},</div><div class="line" id="LC75"> {state,290884,</div>
<div class="line" id="LC76"> "/var/lib/rabbitmq/mnesia/s-rabbit1/msg_store_transient"},</div><div class="line" id="LC77"> rabbit_msg_store_ets_index,</div>
<div class="line" id="LC78">
"/var/lib/rabbitmq/mnesia/s-rabbit1/msg_store_transient",</div><div class="line" id="LC79"> <0.241.0>,294981,286787,299078,303175}},</div><div class="line" id="LC80">
true,0,#Fun<rabbit_mirror_queue_slave.3.29941254>,0,</div><div class="line" id="LC81"> 1,infinity,0,0,0,0,0,</div><div class="line" id="LC82"> {rates,</div>
<div class="line" id="LC83"> {{1341,329623,60054},2},</div><div class="line" id="LC84"> {{1341,329623,60054},2},</div><div class="line" id="LC85"> 1.8427213725389851e-4,1.8427213725389851e-4,</div>
<div class="line" id="LC86"> {1341,340467,310020}},</div><div class="line" id="LC87"> {0,nil},</div><div class="line" id="LC88"> {0,nil},</div><div class="line" id="LC89">
{0,nil},</div><div class="line" id="LC90"> {0,nil},</div><div class="line" id="LC91"> 0,0,</div><div class="line" id="LC92"> {rates,</div>
<div class="line" id="LC93"> {{1341,329623,60054},0},</div><div class="line" id="LC94"> {{1341,329623,60054},2},</div><div class="line" id="LC95"> 0.0,1.8427213725389851e-4,</div>
<div class="line" id="LC96"> {1341,340467,310020}}},</div><div class="line" id="LC97"> undefined,undefined,</div><div class="line" id="LC98"> {dict,2,16,16,8,80,48,</div>
<div class="line" id="LC99"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div class="line" id="LC100"> {{[],[],</div><div class="line" id="LC101"> [[<7471.28937.766>|</div>
<div class="line" id="LC102"> {{[],[]},</div><div class="line" id="LC103"> {set,0,16,16,8,80,48,</div><div class="line" id="LC104"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div class="line" id="LC105"> []},</div><div class="line" id="LC106"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div class="line" id="LC107"> [],[]}}}}]],</div>
<div class="line" id="LC108"> [],</div><div class="line" id="LC109"> [[<7471.28955.766>|</div><div class="line" id="LC110"> {{[],[]},</div>
<div class="line" id="LC111"> {set,0,16,16,8,80,48,</div><div class="line" id="LC112"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div class="line" id="LC113">
[]},</div><div class="line" id="LC114"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div class="line" id="LC115"> [],[]}}}}]],</div>
<div class="line" id="LC116"> [],[],[],[],[],[],[],[],[],[],[]}}},</div><div class="line" id="LC117"> {dict,1,16,16,8,80,48,</div><div class="line" id="LC118"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div class="line" id="LC119"> {{[],[],</div><div class="line" id="LC120"> [[<<8,106,184,177,219,146,140,107,196,247,218,34,</div><div class="line" id="LC121"> 138,15,137,90>>|</div>
<div class="line" id="LC122"> {9,8}]],</div><div class="line" id="LC123"> [],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div><div class="line" id="LC124"> 10,</div>
<div class="line" id="LC125"> {dict,0,16,16,8,80,48,</div><div class="line" id="LC126"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div class="line" id="LC127">
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div><div class="line" id="LC128"> {dict,2,16,16,8,80,48,</div><div class="line" id="LC129"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div class="line" id="LC130"> {{[],[],</div><div class="line" id="LC131"> [[<7471.28937.766>|#Ref<0.0.149.86308>]],</div><div class="line" id="LC132"> [],</div>
<div class="line" id="LC133"> [[<7471.28955.766>|#Ref<0.0.149.81297>]],</div><div class="line" id="LC134"> [],[],[],[],[],[],[],[],[],[],[]}}},</div><div class="line" id="LC135">
true}</div><div class="line" id="LC136">** Reason for termination == </div><div class="line" id="LC137">** {function_clause,</div><div class="line" id="LC138"> [{rabbit_variable_queue,fold,</div>
<div class="line" id="LC139"> ["\b",undefined,</div><div class="line" id="LC140"> {vqstate,</div><div class="line" id="LC141"> {0,{[],[]}},</div><div class="line" id="LC142">
{0,{[],[]}},</div><div class="line" id="LC143"> {delta,undefined,0,undefined},</div><div class="line" id="LC144"> {0,{[],[]}},</div><div class="line" id="LC145"> {0,{[],[]}},</div>
<div class="line" id="LC146"> 9,</div><div class="line" id="LC147"> {1,</div><div class="line" id="LC148"> {8,</div><div class="line" id="LC149"> {msg_status,8,</div>
<div class="line" id="LC150"> <<8,106,184,177,219,146,140,107,196,247,218,34,138,15,</div><div class="line" id="LC151"> 137,90>>,</div><div class="line" id="LC152">
undefined,true,true,true,true,</div><div class="line" id="LC153"> {message_properties,undefined,false}},</div><div class="line" id="LC154"> nil,nil}},</div><div class="line" id="LC155">
undefined,</div><div class="line" id="LC156"> {0,nil},</div><div class="line" id="LC157"> {qistate,</div><div class="line" id="LC158"> "/var/lib/rabbitmq/mnesia/s-rabbit1/queues/6SK5DU9WSNS31F15H65PWTGEF",</div>
<div class="line" id="LC159"> {{dict,0,16,16,8,80,48,</div><div class="line" id="LC160"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div class="line" id="LC161"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div class="line" id="LC162"> [{segment,0,</div><div class="line" id="LC163"> "/var/lib/rabbitmq/mnesia/s-rabbit1/queues/6SK5DU9WSNS31F15H65PWTGEF/0.idx",</div><div class="line" id="LC164">
{array,16384,0,undefined,</div><div class="line" id="LC165"> {{{{{undefined,undefined,undefined,undefined,</div><div class="line" id="LC166"> undefined,undefined,undefined,</div>
<div class="line" id="LC167"> undefined,</div><div class="line" id="LC168"> {{<<8,106,184,177,219,146,140,107,196,</div><div class="line" id="LC169">
247,218,34,138,15,137,90>>,</div><div class="line" id="LC170"> {message_properties,undefined,true},</div><div class="line" id="LC171">
true},</div><div class="line" id="LC172"> del,no_ack},</div><div class="line" id="LC173"> undefined},</div>
<div class="line" id="LC174"> 10,10,10,10,10,10,10,10,10,10},</div><div class="line" id="LC175"> 100,100,100,100,100,100,100,100,100,100},</div><div class="line" id="LC176">
1000,1000,1000,1000,1000,1000,1000,1000,1000,</div><div class="line" id="LC177"> 1000},</div><div class="line" id="LC178"> 10000,10000,10000,10000,10000,10000,10000,</div>
<div class="line" id="LC179"> 10000,10000,10000}},</div><div class="line" id="LC180"> 1}]},</div><div class="line" id="LC181"> #Ref<0.0.113.112114>,2,262144,</div>
<div class="line" id="LC182"> #Fun<rabbit_variable_queue.2.91259159>,</div><div class="line" id="LC183"> {0,nil}},</div><div class="line" id="LC184"> {{client_msstate,msg_store_persistent,</div>
<div class="line" id="LC185"> <<45,226,10,206,192,135,238,119,205,177,43,46,24,159,170,</div><div class="line" id="LC186"> 52>>,</div><div class="line" id="LC187"> {dict,0,16,16,8,80,48,</div>
<div class="line" id="LC188"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div class="line" id="LC189"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div class="line" id="LC190"> {state,311369,</div><div class="line" id="LC191"> "/var/lib/rabbitmq/mnesia/s-rabbit1/msg_store_persistent"},</div><div class="line" id="LC192">
rabbit_msg_store_ets_index,</div><div class="line" id="LC193"> "/var/lib/rabbitmq/mnesia/s-rabbit1/msg_store_persistent",</div><div class="line" id="LC194"> <0.246.0>,315466,307272,319563,323660},</div>
<div class="line" id="LC195"> {client_msstate,msg_store_transient,</div><div class="line" id="LC196"> <<172,50,239,59,161,4,42,98,116,42,171,120,108,181,1,76>>,</div><div class="line" id="LC197">
{dict,0,16,16,8,80,48,</div><div class="line" id="LC198"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div class="line" id="LC199"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div class="line" id="LC200"> {state,290884,</div><div class="line" id="LC201"> "/var/lib/rabbitmq/mnesia/s-rabbit1/msg_store_transient"},</div><div class="line" id="LC202">
rabbit_msg_store_ets_index,</div><div class="line" id="LC203"> "/var/lib/rabbitmq/mnesia/s-rabbit1/msg_store_transient",</div><div class="line" id="LC204"> <0.241.0>,294981,286787,299078,303175}},</div>
<div class="line" id="LC205"> true,0,#Fun<rabbit_mirror_queue_slave.3.29941254>,0,1,</div><div class="line" id="LC206"> infinity,0,0,0,0,0,</div><div class="line" id="LC207"> {rates,</div>
<div class="line" id="LC208"> {{1341,329623,60054},2},</div><div class="line" id="LC209"> {{1341,329623,60054},2},</div><div class="line" id="LC210"> 1.8427213725389851e-4,1.8427213725389851e-4,</div>
<div class="line" id="LC211"> {1341,340467,310020}},</div><div class="line" id="LC212"> {0,nil},</div><div class="line" id="LC213"> {0,nil},</div><div class="line" id="LC214">
{0,nil},</div><div class="line" id="LC215"> {0,nil},</div><div class="line" id="LC216"> 0,0,</div><div class="line" id="LC217"> {rates,</div><div class="line" id="LC218">
{{1341,329623,60054},0},</div><div class="line" id="LC219"> {{1341,329623,60054},2},</div><div class="line" id="LC220"> 0.0,1.8427213725389851e-4,</div><div class="line" id="LC221">
{1341,340467,310020}}}]},</div><div class="line" id="LC222"> {rabbit_mirror_queue_slave,process_instruction,2},</div><div class="line" id="LC223"> {rabbit_mirror_queue_slave,handle_cast,2},</div>
<div class="line" id="LC224"> {gen_server2,handle_msg,2},</div><div class="line" id="LC225"> {proc_lib,wake_up,3}]}</div></pre></div><br clear="all"><br>-- <br>Brian K. Jones<br>My Blog <a href="http://www.protocolostomy.com">http://www.protocolostomy.com</a><br>
Follow me <a href="http://twitter.com/bkjones">http://twitter.com/bkjones</a><br>