<div dir="ltr"><div>Hi,</div><div><br></div><div>I'm running two RabbitMQ nodes in a cluster on Ubuntu 12.04. I have a virtual host there and a ha policy for a queue. Both nodes are disc nodes. During the last 24 hours RabbitMQ crashed twice. Producers are able to continue publishing messages but they don't end up routed or something. rabbitmqctl list_queues does not list the queue. In the HTTP management console the queue is listed but the page only shows:</div>
<div><span style="color:rgb(72,72,72);font-family:Verdana,sans-serif;font-size:12px"><br></span></div><div><span style="color:rgb(72,72,72);font-family:Verdana,sans-serif;font-size:12px">TypeError: Cannot read property 'ram_msg_count' of undefined</span><br>
</div><div><span style="color:rgb(72,72,72);font-family:Verdana,sans-serif;font-size:12px">TypeError: Cannot read property 'ram_msg_count' of undefined</span> </div><div><br></div><div>There should not anything special going on on the servers. Also, they are not under heavy load or something.</div>
<div><br></div><div>After the last crash the log file on both nodes were empty. Unfortunately I didn't backup the logs after the first crashes. I was able to find the following information in xxx-sasl.log:</div><div><br>
</div><div>=CRASH REPORT==== 19-Nov-2013::04:17:22 ===</div><div> crasher:</div><div> initial call: gen:init_it/6</div><div> pid: <0.259.0></div><div> registered_name: []</div><div> exception exit: {{badmatch,{error,not_found}},</div>
<div> [{rabbit_mirror_queue_master,stop_all_slaves,2},</div><div> {rabbit_mirror_queue_master,delete_and_terminate,2},</div><div> {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6},</div>
<div> {rabbit_amqqueue_process,terminate_shutdown,2},</div><div> {gen_server2,terminate,3},</div><div> {proc_lib,wake_up,3}]}</div><div> in function gen_server2:terminate/3</div>
<div> ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.192.0>]</div><div> messages: []</div><div> links: [<0.257.0>,<0.452.0>,#Port<0.13309>]</div><div> dictionary: [{delegate,delegate_15},</div>
<div> {{credit_to,<0.788.0>},20},</div><div> {{#Ref<0.0.0.1271>,fhc_handle},</div><div> {handle,{file_descriptor,prim_file,{#Port<0.13309>,20}},</div><div>
774680,false,0,infinity,[],true,</div><div> "/var/lib/rabbitmq/mnesia/rabbit@mq1-api-test/queues/B1433I2VYA7YIS4WJY5I6NO4R/journal.jif",</div><div> [write,binary,raw,read],</div>
<div> [{write_buffer,infinity}],</div><div> true,true,</div><div> {1384,827437,174496}}},</div><div> {{credit_to,<0.563.0>},1},</div>
<div> {fhc_age_tree,{1,</div><div> {{1384,827437,174496},</div><div> #Ref<0.0.0.1271>,nil,nil}}},</div><div> {{"/var/lib/rabbitmq/mnesia/rabbit@mq1-api-test/queues/B1433I2VYA7YIS4WJY5I6NO4R/journal.jif",</div>
<div> fhc_file},</div><div><div> {file,1,true}},</div><div> {credit_blocked,[]},</div><div> {{ch,<0.8244.0>},</div><div> {cr,<0.8244.0>,#Ref<0.0.10.78984>,</div>
<div> {[],[]},</div><div> 1,</div><div> {queue,[],[],0},</div><div> {qstate,<0.8243.0>,active,{0,nil}},</div><div> 25}},</div>
<div> {{ch,<0.8274.0>},</div><div> {cr,<0.8274.0>,#Ref<0.0.10.91831>,</div><div> {[],[]},</div><div> 1,</div><div> {queue,[],[],0},</div>
<div> {qstate,<0.8273.0>,active,{0,nil}},</div><div> 49}},</div><div> {{credit_to,<0.13668.0>},41},</div><div> {guid,{{1330803274,3383937130,2004047782,2434038047},1}},</div>
<div> {{credit_from,<0.251.0>},1826}]</div><div> trap_exit: true</div><div> status: running</div><div> heap_size: 10946</div><div> stack_size: 24</div><div> reductions: 1514324983</div>
<div> neighbours:</div><div> neighbour: [{pid,<0.453.0>},</div><div> {registered_name,[]},</div><div> {initial_call,{gen,init_it,</div><div> ['Argument__1','Argument__2',</div>
<div> 'Argument__3','Argument__4',</div><div> 'Argument__5','Argument__6']}},</div><div> {current_function,{gen_server2,process_next_msg,1}},</div>
<div> {ancestors,[<0.452.0>,<0.259.0>,rabbit_amqqueue_sup,</div><div> rabbit_sup,<0.192.0>]},</div><div> {messages,[]},</div><div> {links,[<0.452.0>]},</div>
</div><div><div> {dictionary,[{random_seed,{24781,9822,29460}}]},</div><div> {trap_exit,false},</div><div> {status,waiting},</div><div> {heap_size,610},</div>
<div> {stack_size,7},</div><div> {reductions,660407699}]</div><div> neighbour: [{pid,<0.452.0>},</div><div> {registered_name,[]},</div><div> {initial_call,{gen,init_it,</div>
<div> ['Argument__1','Argument__2',</div><div> 'Argument__3','Argument__4',</div><div> 'Argument__5','Argument__6']}},</div>
<div> {current_function,{gen_server2,process_next_msg,1}},</div><div> {ancestors,[<0.259.0>,rabbit_amqqueue_sup,rabbit_sup,</div><div> <0.192.0>]},</div>
<div> {messages,[]},</div><div> {links,[<0.259.0>,<0.453.0>]},</div><div> {dictionary,[]},</div><div> {trap_exit,false},</div><div> {status,waiting},</div>
<div> {heap_size,2584},</div><div> {stack_size,7},</div><div> {reductions,59821}]</div></div><div><br></div><div>=SUPERVISOR REPORT==== 19-Nov-2013::04:17:22 ===</div><div>
Supervisor: {local,rabbit_amqqueue_sup}</div><div> Context: child_terminated</div><div> Reason: {{badmatch,{error,not_found}},</div><div> [{rabbit_mirror_queue_master,stop_all_slaves,2},</div>
<div> {rabbit_mirror_queue_master,delete_and_terminate,2},</div><div> {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6},</div><div> {rabbit_amqqueue_process,terminate_shutdown,2},</div>
<div> {gen_server2,terminate,3},</div><div> {proc_lib,wake_up,3}]}</div><div> Offender: [{pid,<0.259.0>},</div><div> {name,rabbit_amqqueue},</div><div> {mfargs,{rabbit_amqqueue_process,start_link,undefined}},</div>
<div> {restart_type,temporary},</div><div> {shutdown,4294967295},</div><div> {child_type,worker}]</div><div><br></div><div><br></div><div>=CRASH REPORT==== 19-Nov-2013::04:30:20 ===</div>
<div> crasher:</div><div> initial call: gen:init_it/6</div><div> pid: <0.13842.0></div><div> registered_name: []</div><div> exception exit: {{badmatch,true},</div><div> [{rabbit_queue_index,init,2},</div>
<div> {rabbit_variable_queue,init,5},</div><div> {rabbit_mirror_queue_master,init,3},</div><div> {rabbit_amqqueue_process,declare,3},</div><div> {gen_server2,handle_msg,2},</div>
<div> {proc_lib,init_p_do_apply,3}]}</div><div> in function gen_server2:terminate/3</div><div> ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.192.0>]</div><div> messages: []</div><div>
links: [<0.257.0>]</div><div> dictionary: [{{xtype_to_module,direct},rabbit_exchange_type_direct}]</div><div> trap_exit: true</div><div> status: running</div><div> heap_size: 1597</div><div> stack_size: 24</div>
<div> reductions: 1224</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 19-Nov-2013::04:30:20 ===</div><div> Supervisor: {local,rabbit_amqqueue_sup}</div><div> Context: child_terminated</div>
<div> Reason: {{badmatch,true},</div><div> [{rabbit_queue_index,init,2},</div><div> {rabbit_variable_queue,init,5},</div><div> {rabbit_mirror_queue_master,init,3},</div>
<div> {rabbit_amqqueue_process,declare,3},</div><div> {gen_server2,handle_msg,2},</div><div> {proc_lib,init_p_do_apply,3}]}</div><div> Offender: [{pid,<0.13842.0>},</div>
<div> {name,rabbit_amqqueue},</div><div> {mfargs,{rabbit_amqqueue_process,start_link,undefined}},</div><div> {restart_type,temporary},</div><div> {shutdown,4294967295},</div>
<div> {child_type,worker}]</div><div><br></div><div><br></div><div><br></div><div>I have no clue what's going on here. Can anyone give me a hint?</div><div><br></div><div>Best Regards,</div><div>Lauri</div>
</div>