[rabbitmq-discuss] Rabbitmq 2.6.1 crash on CentOS
Simon MacMullen
simon at rabbitmq.com
Tue Jul 23 13:00:35 BST 2013
It looks like the message store crashed, although you haven't posted the
error message from the message store crash, just one of the queues
crashing afterwards because it couldn't find it.
Having said that, a huge number of bugs have been fixed since 2.6.1; an
upgrade is strongly recommended.
Cheers, Simon
On 23/07/13 06:47, mail.zhang.yee wrote:
> Hi All
> Nice to see you here, we recently got a problem during openstack
> deployment in CentOS6.3.
> The rabbitmq constantly crash after running one or two days.
> Cloud somebody here can take a look at what's happened inside rabbitmq
> and how to fix it?
> detailed status and logs as below. thanks in advance.
> ======================================
> rabbitmqctl status
> Status of node rabbit at ci91szcmp001 ...
> [{pid,29272},
> {running_applications,[{rabbit,"RabbitMQ","2.6.1"},
> {os_mon,"CPO CXC 138 46","2.2.7"},
> {sasl,"SASL CXC 138 11","2.1.10"},
> {mnesia,"MNESIA CXC 138 12","4.5"},
> {stdlib,"ERTS CXC 138 10","1.17.5"},
> {kernel,"ERTS CXC 138 10","2.14.5"}]},
> {os,{unix,linux}},
> {erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit]
> [smp:32:32] [rq:32] [async-threads:30] [kernel-poll:true]\n"},
> {memory,[{total,74666016},
> {processes,45884208},
> {processes_used,45866296},
> {system,28781808},
> {atom,1119593},
> {atom_used,1113078},
> {binary,3421464},
> {code,11053747},
> {ets,7607264}]}]
> ...done.
> ======================================
> =ERROR REPORT==== 22-Jul-2013::14:50:26 ===
> ** Generic server <0.1921.0> terminating
> ** Last message in was {'EXIT',<0.237.0>,shutdown}
> ** When Server state == {q,
> {amqqueue,
> {resource,<<"/">>,queue,
> <<"conductor.ci91szcmp004.webex.com
> <http://conductor.ci91szcmp004.webex.com>">>},
> false,false,none,
> [{<<"x-ha-policy">>,longstr,<<"all">>}],
> <0.1921.0>,[],all},
> none,true,rabbit_mirror_queue_master,
>
> {state,<0.1923.0>,<0.1922.0>,rabbit_variable_queue,
> {vqstate,
> {[],[]},
> {0,{[],[]}},
> {delta,undefined,0,undefined},
> {0,{[],[]}},
> {[],[]},
> 0,
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}},
> undefined,
> {0,nil},
> {qistate,
>
> "/var/lib/rabbitmq/mnesia/rabbit at ci91szcmp001/queues/A1FBAI3711PERTSXW5DHR0X8Y",
> {{dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}},
> []},
> undefined,0,262144,
> #Fun<rabbit_variable_queue.2.41165857>,[]},
> {undefined,
> {client_msstate,msg_store_transient,
>
> <<207,96,115,58,236,124,180,246,67,18,228,236,182,
> 66,117,84>>,
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}},
> {state,200768,
>
> "/var/lib/rabbitmq/mnesia/rabbit at ci91szcmp001/msg_store_transient"},
> rabbit_msg_store_ets_index,
>
> "/var/lib/rabbitmq/mnesia/rabbit at ci91szcmp001/msg_store_transient",
> <0.228.0>,204865,196671,208962}},
>
> false,0,#Fun<rabbit_amqqueue_process.5.31190493>,0,
> 0,infinity,0,0,0,0,0,0,
> {rates,
> {{1374,213408,350099},0},
> {{1374,213408,350099},0},
> 0.0,0.0,
> {1374,213451,550036}},
> {0,nil},
> {0,nil},
> {0,nil},
> {0,nil},
> 0,0,
> {rates,
> {{1374,213408,350099},0},
> {{1374,213408,350099},0},
> 0.0,0.0,
> {1374,213451,550036}}},
> 0,
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}},
> [],
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}},
> {set,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}}},
> {[],[]},
> {[],[]},
>
> undefined,undefined,#Ref<0.0.742.237144>,undefined,
> {state,none,5000,undefined},
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
> undefined,undefined}
> ** Reason for termination ==
> ** {noproc,
> {gen_server2,call,
> [msg_store_transient,
> {client_terminate,
>
> <<207,96,115,58,236,124,180,246,67,18,228,236,182,66,117,84>>},
> infinity]}}
> ** In 'terminate' callback with reason ==
> ** shutdown
>
> =ERROR REPORT==== 22-Jul-2013::14:50:26 ===
> ** gen_event handler rabbit_error_logger crashed.
> ** Was installed in error_logger
> ** Last event was: {error,<0.119.0>,
> {<0.1887.0>,
> "** Generic server ~p terminating~n** Last message
> in was ~p~n** When Server state == ~p~n** Reason for termination == ~n**
> ~p~n** In 'terminate' callback with reason ==~n** ~p~n",
> [<0.1887.0>,
> {'EXIT',<0.237.0>,shutdown},
> {q,
> {amqqueue,
> {resource,<<"/">>,queue,<<"consoleauth">>},
> false,false,none,
> [{<<"x-ha-policy">>,longstr,<<"all">>}],
> <0.1887.0>,[],all},
> none,true,rabbit_mirror_queue_master,
> {state,<0.1889.0>,<0.1888.0>,rabbit_variable_queue,
> {vqstate,
> {[],[]},
> {0,{[],[]}},
> {delta,undefined,0,undefined},
> {0,{[],[]}},
> {[],[]},
> 8,
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
> undefined,
> {0,nil},
> {qistate,
>
> "/var/lib/rabbitmq/mnesia/rabbit at ci91szcmp001/queues/1AV7T4P6H9GB74RNTGRVPV2OY",
> {{dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}},
> []},
> undefined,0,262144,
> #Fun<rabbit_variable_queue.2.41165857>,[]},
> {undefined,
> {client_msstate,msg_store_transient,
> <<"ܺ�o$��W����\b?J">>,
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> []}}},
> {state,200768,
>
> "/var/lib/rabbitmq/mnesia/rabbit at ci91szcmp001/msg_store_transient"},
> rabbit_msg_store_ets_index,
>
> "/var/lib/rabbitmq/mnesia/rabbit at ci91szcmp001/msg_store_transient",
> <0.228.0>,204865,196671,208962}},
>
> false,0,#Fun<rabbit_amqqueue_process.5.31190493>,0,0,
> infinity,0,0,0,0,0,0,
> {rates,
> {{1374,460116,880026},2},
> {{1374,460116,880026},2},
> 0.0016008717049531846,0.0016008717049531846,
> {1374,461800,598037}},
> {0,nil},
> {0,nil},
> {0,nil},
> {0,nil},
> 0,0,
> {rates,
> {{1374,460116,880026},2},
> {{1374,460116,880026},2},
> 0.0016008717049531846,0.0016008717049531846,
> {1374,461800,598037}}},
> 0,
> {dict,0,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
> [],
> {dict,0,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
> {set,0,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
> {[],[]},
> {[],[]},
> undefined,undefined,#Ref<0.0.742.237570>,undefined,
> {state,none,5000,undefined},
> {dict,0,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
> undefined,undefined},
> {noproc,
> {gen_server2,call,
> [msg_store_transient,
> {client_terminate,<<"ܺ�o$��W����\b?J">>},
> infinity]}},
> shutdown]}}
> ** When handler state == {resource,<<"/">>,exchange,<<"amq.rabbitmq.log">>}
> ** Reason == {badarg,[{ets,lookup,[rabbit_registry,{exchange,topic}]},
> {rabbit_registry,lookup_module,2},
> {rabbit_exchange,type_to_module,1},
> {rabbit_exchange,route1,2},
> {rabbit_basic,publish,2},
> {rabbit_error_logger,publish1,4},
> {rabbit_error_logger,handle_event,2},
> {gen_event,server_update,4}]}
>
> =INFO REPORT==== 22-Jul-2013::14:50:27 ===
> application: rabbit
> exited: shutdown
> type: permanent
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
--
Simon MacMullen
RabbitMQ, Pivotal
More information about the rabbitmq-discuss
mailing list