One of my rabbitmq boxes (with bug21673 branc) stopped responding and dumped a boat load of crash reports to rabbit-sasl.log. Below are the last four crash reports excerpted from the file. Two questions: <br><br>1) when I stopped and restarted rabbitmq, after quite some time only half about half the queues are showing up, the other half haven't showed up after several hours, is there a way to recover the other queues and the data associated with them, also <br>
<br>2) Any ideas from these crash reports what might be going on?<br><br>thanks,<br>Scott <br><br>=CRASH REPORT==== 1-Mar-2010::23:25:56 ===<br> crasher:<br> initial call: gen:init_it/6<br> pid: <0.26578.21><br>
registered_name: []<br> exception exit: {{{badarg,<br> [{ets,lookup,<br> [rabbit_msg_store_cache,<br> <<184,164,57,196,186,77,197,52,166,133,102,<br>
139,114,25,127,109>>]},<br> {rabbit_msg_store,fetch_and_increment_cache,1},<br> {rabbit_msg_store,read,2},<br> {rabbit_variable_queue,fetch_from_q3_or_delta,1},<br>
{rabbit_amqqueue_process,handle_call,3},<br> {gen_server2,handle_msg,7},<br> {proc_lib,wake_up,3}]},<br> {gen_server2,call,<br>
[<0.4306.0>,<br> {basic_get,<0.26578.21>,false},<br> infinity]}},<br> [{gen_server2,call,3},<br> {rabbit_misc,with_exit_handler,2},<br>
{rabbit_channel,handle_method,3},<br> {rabbit_channel,handle_cast,2},<br> {gen_server2,handle_msg,7},<br> {proc_lib,wake_up,3}]}<br> in function gen_server2:terminate/6<br>
ancestors: [<0.26577.21>]<br> messages: []<br> links: [<0.26576.21>,<0.26577.21>]<br> dictionary: [{permission_cache,<br> [{{resource,<<"/">>,queue,<<"scoring-completed">>},<br>
read},<br> {{resource,<<"/">>,exchange,<<"scoring">>},write}]},<br> {guid,{{4,<0.26578.21>},920577}}]<br> trap_exit: true<br>
status: running<br> heap_size: 121393<br> stack_size: 24<br> reductions: 330830447<br> neighbours:<br> neighbour: [{pid,<0.26576.21>},<br> {registered_name,[]},<br> {initial_call,{rabbit_writer,mainloop,1}},<br>
{current_function,{erlang,hibernate,3}},<br> {ancestors,[]},<br> {messages,[shutdown]},<br> {links,[<0.26578.21>]},<br> {dictionary,[]},<br>
{trap_exit,false},<br> {status,runnable},<br> {heap_size,7},<br> {stack_size,0},<br> {reductions,13080564}]<br><br><br>=CRASH REPORT==== 1-Mar-2010::23:46:28 ===<br>
crasher:<br> initial call: gen:init_it/6<br> pid: <0.1518.0><br> registered_name: []<br> exception exit: {badarg,[{ets,lookup,<br> [217134,<br> <<6,83,78,49,79,166,176,26,232,128,113,<br>
103,68,127,144,203>>]},<br> {rabbit_msg_store_ets_index,lookup,2},<br> {rabbit_msg_store,read,2},<br> {rabbit_variable_queue,fetch_from_q3_or_delta,1},<br>
{rabbit_amqqueue_process,handle_call,3},<br> {gen_server2,handle_msg,7},<br> {proc_lib,wake_up,3}]}<br> in function gen_server2:terminate/6<br>
ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.101.0>]<br> messages: []<br> links: [<0.1267.0>]<br> dictionary: [{fhc_age_tree,{0,nil}}]<br> trap_exit: true<br> status: running<br> heap_size: 317811<br>
stack_size: 24<br> reductions: 106370888<br> neighbours:<br><br>=CRASH REPORT==== 1-Mar-2010::23:46:28 ===<br> crasher:<br> initial call: gen:init_it/6<br> pid: <0.9205.23><br> registered_name: []<br>
exception exit: {{{badarg,<br> [{ets,lookup,<br> [217134,<br> <<6,83,78,49,79,166,176,26,232,128,113,103,<br> 68,127,144,203>>]},<br>
{rabbit_msg_store_ets_index,lookup,2},<br> {rabbit_msg_store,read,2},<br> {rabbit_variable_queue,fetch_from_q3_or_delta,1},<br> {rabbit_amqqueue_process,handle_call,3},<br>
{gen_server2,handle_msg,7},<br> {proc_lib,wake_up,3}]},<br> {gen_server2,call,<br> [<0.1518.0>,<br> {basic_get,<0.9205.23>,false},<br>
infinity]}},<br> [{gen_server2,call,3},<br> {rabbit_misc,with_exit_handler,2},<br> {rabbit_channel,handle_method,3},<br> {rabbit_channel,handle_cast,2},<br>
{gen_server2,handle_msg,7},<br> {proc_lib,wake_up,3}]}<br> in function gen_server2:terminate/6<br> ancestors: [<0.9204.23>]<br> messages: []<br> links: [<0.9203.23>,<0.9204.23>]<br>
dictionary: [{permission_cache,<br> [{{resource,<<"/">>,queue,<<"update-HIGH">>},<br> read}]},<br> {guid,{{4,<0.9205.23>},94}}]<br>
trap_exit: true<br> status: running<br> heap_size: 610<br> stack_size: 24<br> reductions: 47972<br> neighbours:<br> neighbour: [{pid,<0.9203.23>},<br> {registered_name,[]},<br> {initial_call,{rabbit_writer,mainloop,1}},<br>
{current_function,{erlang,hibernate,3}},<br> {ancestors,[]},<br> {messages,[shutdown]},<br> {links,[<0.9205.23>]},<br> {dictionary,[]},<br>
{trap_exit,false},<br> {status,runnable},<br> {heap_size,7},<br> {stack_size,0},<br> {reductions,278}]<br><br><br>=CRASH REPORT==== 1-Mar-2010::23:46:28 ===<br>
crasher:<br> initial call: gen:init_it/6<br> pid: <0.9205.23><br> registered_name: []<br> exception exit: {{{badarg,<br> [{ets,lookup,<br> [217134,<br>
<<6,83,78,49,79,166,176,26,232,128,113,103,<br> 68,127,144,203>>]},<br> {rabbit_msg_store_ets_index,lookup,2},<br> {rabbit_msg_store,read,2},<br>
{rabbit_variable_queue,fetch_from_q3_or_delta,1},<br> {rabbit_amqqueue_process,handle_call,3},<br> {gen_server2,handle_msg,7},<br> {proc_lib,wake_up,3}]},<br>
{gen_server2,call,<br> [<0.1518.0>,<br> {basic_get,<0.9205.23>,false},<br> infinity]}},<br> [{gen_server2,call,3},<br>
{rabbit_misc,with_exit_handler,2},<br> {rabbit_channel,handle_method,3},<br> {rabbit_channel,handle_cast,2},<br> {gen_server2,handle_msg,7},<br>
{proc_lib,wake_up,3}]}<br> in function gen_server2:terminate/6<br> ancestors: [<0.9204.23>]<br> messages: []<br> links: [<0.9203.23>,<0.9204.23>]<br> dictionary: [{permission_cache,<br>
[{{resource,<<"/">>,queue,<<"update-HIGH">>},<br> read}]},<br> {guid,{{4,<0.9205.23>},94}}]<br> trap_exit: true<br>
status: running<br> heap_size: 610<br> stack_size: 24<br> reductions: 47972<br> neighbours:<br> neighbour: [{pid,<0.9203.23>},<br> {registered_name,[]},<br> {initial_call,{rabbit_writer,mainloop,1}},<br>
{current_function,{erlang,hibernate,3}},<br> {ancestors,[]},<br> {messages,[shutdown]},<br> {links,[<0.9205.23>]},<br> {dictionary,[]},<br>
{trap_exit,false},<br> {status,runnable},<br> {heap_size,7},<br> {stack_size,0},<br> {reductions,278}]<br><br>