[rabbitmq-discuss] Sudden crash (3.0.2)

[Jason McIntosh]

Got some really funky errors and a sudden crash of my entire cluster.
 Off hand, I'm GUESSING it's a disk error, but I'm not totally sure -
thought I'd see if anyone had any ideas?
Thanks!
Jason


The web ui returns the following:

{error,{error,badarg,
              [{ets,match,[rabbit_registry,{{exchange,'$1'},'$2'}],[]},
               {rabbit_registry,lookup_all,1,[]},
               {rabbit_mgmt_external_stats,list_registry_plugins,2,[]},
               {rabbit_mgmt_wm_overview,to_json,2,[]},
               {webmachine_resource,resource_call,3,[]},
               {webmachine_resource,do,3,[]},
               {webmachine_decision_core,resource_call,1,[]},
               {webmachine_decision_core,decision,1,[]}]}}


In the cluster.log file I see a TON of data and some really odd things like:

  {msg_status,198128,
                               <<8,75,158,13,139,133,182,107,227,74,148,135,78,
                                 181,221,89>>,
                               undefined,true,false,true,true,
                               {message_properties,undefined,true,false}},
                              {msg_status,198127,<<"¯?VõFþCTÃ
0k2­ÿIÃ">>,                  undefined,true,false,true,true,
                               {message_properties,undefined,true,false}},
                              {msg_status,198126,
                               <<207,120,238,6,2,192,31,43,235,1,10,138,104,
                                 207,34,236>>,


=ERROR REPORT==== 14-Aug-2013::09:53:50 ===
** Generic server rabbit_mgmt_external_stats terminating
** Last message in was emit_update
** When Server state == {state,1024}
** Reason for termination ==
** {noproc,{gen_server,call,[rabbit_node_monitor,partitions,infinity]}}

** Reason for termination ==
** {{badmatch,{error,eio}},
    [{file_handle_cache,soft_close,1,[]},
     {file_handle_cache,hard_close,1,[]},
     {file_handle_cache,close,1,[]},
     {rabbit_msg_store,terminate,2,[]},
     {gen_server2,terminate,3,[]},
     {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}
** In 'terminate' callback with reason ==
** shutdown




=ERROR REPORT==== 14-Aug-2013::09:53:50 ===
** Generic server msg_store_transient terminating
** Last message in was {'EXIT',<0.238.0>,shutdown}
** When Server state == {msstate,

"/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_transient",
                            rabbit_msg_store_ets_index,
                            {state,397381,

"/data/rabbitmq/rabbitmq/mnesia/cluster/msg_store_transient"},
                            0,#Ref<0.0.0.1839>,
                            {dict,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            undefined,0,1271,[],<0.291.0>,401478,393277,
                            405575,409672,
                            {set,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            {dict,48,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},




-- 
Jason McIntosh
http://mcintosh.poetshome.com/blog/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130814/2032e250/attachment.htm>