<div dir="ltr">Hi All <div>Nice to see you here, we recently got a problem during openstack deployment in CentOS6.3.</div><div>The rabbitmq constantly crash after running one or two days. </div><div>Cloud somebody here can take a look at what's happened inside rabbitmq</div>
<div>and how to fix it?</div><div>detailed status and logs as below. thanks in advance. </div><div><div>======================================</div><div> rabbitmqctl status</div><div>Status of node rabbit@ci91szcmp001 ...</div>
<div>[{pid,29272},</div><div> {running_applications,[{rabbit,"RabbitMQ","2.6.1"},</div><div> {os_mon,"CPO CXC 138 46","2.2.7"},</div><div> {sasl,"SASL CXC 138 11","2.1.10"},</div>
<div> {mnesia,"MNESIA CXC 138 12","4.5"},</div><div> {stdlib,"ERTS CXC 138 10","1.17.5"},</div><div> {kernel,"ERTS CXC 138 10","2.14.5"}]},</div>
<div> {os,{unix,linux}},</div><div> {erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:32:32] [rq:32] [async-threads:30] [kernel-poll:true]\n"},</div><div> {memory,[{total,74666016},</div><div> {processes,45884208},</div>
<div> {processes_used,45866296},</div><div> {system,28781808},</div><div> {atom,1119593},</div><div> {atom_used,1113078},</div><div> {binary,3421464},</div><div> {code,11053747},</div>
<div> {ets,7607264}]}]</div><div>...done.</div><div>======================================</div></div><div><div>=ERROR REPORT==== 22-Jul-2013::14:50:26 ===</div><div>** Generic server <0.1921.0> terminating</div>
<div>** Last message in was {'EXIT',<0.237.0>,shutdown}</div><div>** When Server state == {q,</div><div> {amqqueue,</div><div> {resource,<<"/">>,queue,</div>
<div> <<"<a href="http://conductor.ci91szcmp004.webex.com">conductor.ci91szcmp004.webex.com</a>">>},</div><div> false,false,none,</div><div> [{<<"x-ha-policy">>,longstr,<<"all">>}],</div>
<div> <0.1921.0>,[],all},</div><div> none,true,rabbit_mirror_queue_master,</div><div> {state,<0.1923.0>,<0.1922.0>,rabbit_variable_queue,</div>
<div> {vqstate,</div><div> {[],[]},</div><div> {0,{[],[]}},</div><div> {delta,undefined,0,undefined},</div><div> {0,{[],[]}},</div>
<div> {[],[]},</div><div> 0,</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}},</div><div> undefined,</div><div> {0,nil},</div>
<div> {qistate,</div><div> "/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/queues/A1FBAI3711PERTSXW5DHR0X8Y",</div><div> {{dict,0,16,16,8,80,48,</div>
<div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> []}}},</div><div> []},</div><div> undefined,0,262144,</div><div> #Fun<rabbit_variable_queue.2.41165857>,[]},</div>
<div> {undefined,</div><div> {client_msstate,msg_store_transient,</div><div> <<207,96,115,58,236,124,180,246,67,18,228,236,182,</div><div>
66,117,84>>,</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []},</div>
<div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}},</div><div> {state,200768,</div><div> "/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/msg_store_transient"},</div>
<div> rabbit_msg_store_ets_index,</div><div> "/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/msg_store_transient",</div><div> <0.228.0>,204865,196671,208962}},</div>
<div> false,0,#Fun<rabbit_amqqueue_process.5.31190493>,0,</div><div> 0,infinity,0,0,0,0,0,0,</div><div> {rates,</div><div> {{1374,213408,350099},0},</div>
<div> {{1374,213408,350099},0},</div><div> 0.0,0.0,</div><div> {1374,213451,550036}},</div><div> {0,nil},</div><div>
{0,nil},</div><div> {0,nil},</div><div> {0,nil},</div><div> 0,0,</div><div> {rates,</div><div>
{{1374,213408,350099},0},</div><div> {{1374,213408,350099},0},</div><div> 0.0,0.0,</div><div> {1374,213451,550036}}},</div>
<div> 0,</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> []}}},</div><div> [],</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}},</div><div> {set,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}}},</div><div> {[],[]},</div><div> {[],[]},</div><div>
undefined,undefined,#Ref<0.0.742.237144>,undefined,</div><div> {state,none,5000,undefined},</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div><div> undefined,undefined}</div><div>** Reason for termination == </div><div>** {noproc,</div><div> {gen_server2,call,</div>
<div> [msg_store_transient,</div><div> {client_terminate,</div><div> <<207,96,115,58,236,124,180,246,67,18,228,236,182,66,117,84>>},</div><div> infinity]}}</div><div>
** In 'terminate' callback with reason ==</div><div>** shutdown</div><div><br></div><div>=ERROR REPORT==== 22-Jul-2013::14:50:26 ===</div><div>** gen_event handler rabbit_error_logger crashed.</div><div>** Was installed in error_logger</div>
<div>** Last event was: {error,<0.119.0>,</div><div> {<0.1887.0>,</div><div> "** Generic server ~p terminating~n** Last message in was ~p~n** When Server state == ~p~n** Reason for termination == ~n** ~p~n** In 'terminate' callback with reason ==~n** ~p~n",</div>
<div> [<0.1887.0>,</div><div> {'EXIT',<0.237.0>,shutdown},</div><div> {q,</div><div> {amqqueue,</div><div> {resource,<<"/">>,queue,<<"consoleauth">>},</div>
<div> false,false,none,</div><div> [{<<"x-ha-policy">>,longstr,<<"all">>}],</div><div> <0.1887.0>,[],all},</div>
<div> none,true,rabbit_mirror_queue_master,</div><div> {state,<0.1889.0>,<0.1888.0>,rabbit_variable_queue,</div><div> {vqstate,</div><div> {[],[]},</div>
<div> {0,{[],[]}},</div><div> {delta,undefined,0,undefined},</div><div> {0,{[],[]}},</div><div> {[],[]},</div><div> 8,</div>
<div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div> undefined,</div><div> {0,nil},</div><div> {qistate,</div><div> "/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/queues/1AV7T4P6H9GB74RNTGRVPV2OY",</div>
<div> {{dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> []}}},</div><div> []},</div><div> undefined,0,262144,</div><div> #Fun<rabbit_variable_queue.2.41165857>,[]},</div>
<div> {undefined,</div><div> {client_msstate,msg_store_transient,</div><div> <<"ܺ�o$��W����\b?J">>,</div><div> {dict,0,16,16,8,80,48,</div>
<div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> []}}},</div>
<div> {state,200768,</div><div> "/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/msg_store_transient"},</div><div> rabbit_msg_store_ets_index,</div>
<div> "/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/msg_store_transient",</div><div> <0.228.0>,204865,196671,208962}},</div><div> false,0,#Fun<rabbit_amqqueue_process.5.31190493>,0,0,</div>
<div> infinity,0,0,0,0,0,0,</div><div> {rates,</div><div> {{1374,460116,880026},2},</div><div> {{1374,460116,880026},2},</div>
<div> 0.0016008717049531846,0.0016008717049531846,</div><div> {1374,461800,598037}},</div><div> {0,nil},</div><div> {0,nil},</div>
<div> {0,nil},</div><div> {0,nil},</div><div> 0,0,</div><div> {rates,</div><div> {{1374,460116,880026},2},</div>
<div> {{1374,460116,880026},2},</div><div> 0.0016008717049531846,0.0016008717049531846,</div><div> {1374,461800,598037}}},</div><div> 0,</div>
<div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div> [],</div><div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div> {set,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},</div>
<div> {[],[]},</div><div> {[],[]},</div><div> undefined,undefined,#Ref<0.0.742.237570>,undefined,</div><div> {state,none,5000,undefined},</div>
<div> {dict,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div> undefined,undefined},</div><div> {noproc,</div><div> {gen_server2,call,</div><div> [msg_store_transient,</div><div> {client_terminate,<<"ܺ�o$��W����\b?J">>},</div>
<div> infinity]}},</div><div> shutdown]}}</div><div>** When handler state == {resource,<<"/">>,exchange,<<"amq.rabbitmq.log">>}</div>
<div>** Reason == {badarg,[{ets,lookup,[rabbit_registry,{exchange,topic}]},</div><div> {rabbit_registry,lookup_module,2},</div><div> {rabbit_exchange,type_to_module,1},</div><div>
{rabbit_exchange,route1,2},</div><div> {rabbit_basic,publish,2},</div><div> {rabbit_error_logger,publish1,4},</div><div> {rabbit_error_logger,handle_event,2},</div>
<div> {gen_event,server_update,4}]}</div><div><br></div><div>=INFO REPORT==== 22-Jul-2013::14:50:27 ===</div><div> application: rabbit</div><div> exited: shutdown</div><div> type: permanent</div>
</div><div><br></div></div>