<div dir="ltr">Hi All <div>Nice to see you here, we recently got a problem during openstack deployment in CentOS6.3.</div><div>The rabbitmq constantly crash after running one or two days. </div><div>Cloud somebody here can take a look at what&#39;s happened inside rabbitmq</div>
<div>and how to fix it?</div><div>detailed status and logs as below. thanks in advance. </div><div><div>======================================</div><div> rabbitmqctl status</div><div>Status of node rabbit@ci91szcmp001 ...</div>
<div>[{pid,29272},</div><div> {running_applications,[{rabbit,&quot;RabbitMQ&quot;,&quot;2.6.1&quot;},</div><div>                        {os_mon,&quot;CPO  CXC 138 46&quot;,&quot;2.2.7&quot;},</div><div>                        {sasl,&quot;SASL  CXC 138 11&quot;,&quot;2.1.10&quot;},</div>
<div>                        {mnesia,&quot;MNESIA  CXC 138 12&quot;,&quot;4.5&quot;},</div><div>                        {stdlib,&quot;ERTS  CXC 138 10&quot;,&quot;1.17.5&quot;},</div><div>                        {kernel,&quot;ERTS  CXC 138 10&quot;,&quot;2.14.5&quot;}]},</div>
<div> {os,{unix,linux}},</div><div> {erlang_version,&quot;Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:32:32] [rq:32] [async-threads:30] [kernel-poll:true]\n&quot;},</div><div> {memory,[{total,74666016},</div><div>          {processes,45884208},</div>
<div>          {processes_used,45866296},</div><div>          {system,28781808},</div><div>          {atom,1119593},</div><div>          {atom_used,1113078},</div><div>          {binary,3421464},</div><div>          {code,11053747},</div>
<div>          {ets,7607264}]}]</div><div>...done.</div><div>======================================</div></div><div><div>=ERROR REPORT==== 22-Jul-2013::14:50:26 ===</div><div>** Generic server &lt;0.1921.0&gt; terminating</div>
<div>** Last message in was {&#39;EXIT&#39;,&lt;0.237.0&gt;,shutdown}</div><div>** When Server state == {q,</div><div>                         {amqqueue,</div><div>                          {resource,&lt;&lt;&quot;/&quot;&gt;&gt;,queue,</div>
<div>                           &lt;&lt;&quot;<a href="http://conductor.ci91szcmp004.webex.com">conductor.ci91szcmp004.webex.com</a>&quot;&gt;&gt;},</div><div>                          false,false,none,</div><div>                          [{&lt;&lt;&quot;x-ha-policy&quot;&gt;&gt;,longstr,&lt;&lt;&quot;all&quot;&gt;&gt;}],</div>
<div>                          &lt;0.1921.0&gt;,[],all},</div><div>                         none,true,rabbit_mirror_queue_master,</div><div>                         {state,&lt;0.1923.0&gt;,&lt;0.1922.0&gt;,rabbit_variable_queue,</div>
<div>                          {vqstate,</div><div>                           {[],[]},</div><div>                           {0,{[],[]}},</div><div>                           {delta,undefined,0,undefined},</div><div>                           {0,{[],[]}},</div>
<div>                           {[],[]},</div><div>                           0,</div><div>                           {dict,0,16,16,8,80,48,</div><div>                            {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div>                            {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div>                              []}}},</div><div>                           undefined,</div><div>                           {0,nil},</div>
<div>                           {qistate,</div><div>                            &quot;/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/queues/A1FBAI3711PERTSXW5DHR0X8Y&quot;,</div><div>                            {{dict,0,16,16,8,80,48,</div>
<div>                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div>                               []},</div><div>                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div>                                []}}},</div><div>                             []},</div><div>                            undefined,0,262144,</div><div>                            #Fun&lt;rabbit_variable_queue.2.41165857&gt;,[]},</div>
<div>                           {undefined,</div><div>                            {client_msstate,msg_store_transient,</div><div>                             &lt;&lt;207,96,115,58,236,124,180,246,67,18,228,236,182,</div><div>
                               66,117,84&gt;&gt;,</div><div>                             {dict,0,16,16,8,80,48,</div><div>                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div>                               []},</div>
<div>                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div>                                []}}},</div><div>                             {state,200768,</div><div>                              &quot;/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/msg_store_transient&quot;},</div>
<div>                             rabbit_msg_store_ets_index,</div><div>                             &quot;/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/msg_store_transient&quot;,</div><div>                             &lt;0.228.0&gt;,204865,196671,208962}},</div>
<div>                           false,0,#Fun&lt;rabbit_amqqueue_process.5.31190493&gt;,0,</div><div>                           0,infinity,0,0,0,0,0,0,</div><div>                           {rates,</div><div>                            {{1374,213408,350099},0},</div>
<div>                            {{1374,213408,350099},0},</div><div>                            0.0,0.0,</div><div>                            {1374,213451,550036}},</div><div>                           {0,nil},</div><div>
                           {0,nil},</div><div>                           {0,nil},</div><div>                           {0,nil},</div><div>                           0,0,</div><div>                           {rates,</div><div>
                            {{1374,213408,350099},0},</div><div>                            {{1374,213408,350099},0},</div><div>                            0.0,0.0,</div><div>                            {1374,213451,550036}}},</div>
<div>                          0,</div><div>                          {dict,0,16,16,8,80,48,</div><div>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div>                             []}}},</div><div>                          [],</div><div>                          {dict,0,16,16,8,80,48,</div><div>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div>                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div>                             []}}},</div><div>                          {set,0,16,16,8,80,48,</div><div>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div>                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div>                             []}}}},</div><div>                         {[],[]},</div><div>                         {[],[]},</div><div>
                         undefined,undefined,#Ref&lt;0.0.742.237144&gt;,undefined,</div><div>                         {state,none,5000,undefined},</div><div>                         {dict,0,16,16,8,80,48,</div><div>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div>
<div>                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div><div>                         undefined,undefined}</div><div>** Reason for termination == </div><div>** {noproc,</div><div>       {gen_server2,call,</div>
<div>           [msg_store_transient,</div><div>            {client_terminate,</div><div>                &lt;&lt;207,96,115,58,236,124,180,246,67,18,228,236,182,66,117,84&gt;&gt;},</div><div>            infinity]}}</div><div>
** In &#39;terminate&#39; callback with reason ==</div><div>** shutdown</div><div><br></div><div>=ERROR REPORT==== 22-Jul-2013::14:50:26 ===</div><div>** gen_event handler rabbit_error_logger crashed.</div><div>** Was installed in error_logger</div>
<div>** Last event was: {error,&lt;0.119.0&gt;,</div><div>                    {&lt;0.1887.0&gt;,</div><div>                     &quot;** Generic server ~p terminating~n** Last message in was ~p~n** When Server state == ~p~n** Reason for termination == ~n** ~p~n** In &#39;terminate&#39; callback with reason ==~n** ~p~n&quot;,</div>
<div>                     [&lt;0.1887.0&gt;,</div><div>                      {&#39;EXIT&#39;,&lt;0.237.0&gt;,shutdown},</div><div>                      {q,</div><div>                       {amqqueue,</div><div>                        {resource,&lt;&lt;&quot;/&quot;&gt;&gt;,queue,&lt;&lt;&quot;consoleauth&quot;&gt;&gt;},</div>
<div>                        false,false,none,</div><div>                        [{&lt;&lt;&quot;x-ha-policy&quot;&gt;&gt;,longstr,&lt;&lt;&quot;all&quot;&gt;&gt;}],</div><div>                        &lt;0.1887.0&gt;,[],all},</div>
<div>                       none,true,rabbit_mirror_queue_master,</div><div>                       {state,&lt;0.1889.0&gt;,&lt;0.1888.0&gt;,rabbit_variable_queue,</div><div>                        {vqstate,</div><div>                         {[],[]},</div>
<div>                         {0,{[],[]}},</div><div>                         {delta,undefined,0,undefined},</div><div>                         {0,{[],[]}},</div><div>                         {[],[]},</div><div>                         8,</div>
<div>                         {dict,0,16,16,8,80,48,</div><div>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div>                         undefined,</div><div>                         {0,nil},</div><div>                         {qistate,</div><div>                          &quot;/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/queues/1AV7T4P6H9GB74RNTGRVPV2OY&quot;,</div>
<div>                          {{dict,0,16,16,8,80,48,</div><div>                            {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                            {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div>                              []}}},</div><div>                           []},</div><div>                          undefined,0,262144,</div><div>                          #Fun&lt;rabbit_variable_queue.2.41165857&gt;,[]},</div>
<div>                         {undefined,</div><div>                          {client_msstate,msg_store_transient,</div><div>                           &lt;&lt;&quot;ܺ�o$��W����\b?J&quot;&gt;&gt;,</div><div>                           {dict,0,16,16,8,80,48,</div>
<div>                            {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                            {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div>                              []}}},</div>
<div>                           {state,200768,</div><div>                            &quot;/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/msg_store_transient&quot;},</div><div>                           rabbit_msg_store_ets_index,</div>
<div>                           &quot;/var/lib/rabbitmq/mnesia/rabbit@ci91szcmp001/msg_store_transient&quot;,</div><div>                           &lt;0.228.0&gt;,204865,196671,208962}},</div><div>                         false,0,#Fun&lt;rabbit_amqqueue_process.5.31190493&gt;,0,0,</div>
<div>                         infinity,0,0,0,0,0,0,</div><div>                         {rates,</div><div>                          {{1374,460116,880026},2},</div><div>                          {{1374,460116,880026},2},</div>
<div>                          0.0016008717049531846,0.0016008717049531846,</div><div>                          {1374,461800,598037}},</div><div>                         {0,nil},</div><div>                         {0,nil},</div>
<div>                         {0,nil},</div><div>                         {0,nil},</div><div>                         0,0,</div><div>                         {rates,</div><div>                          {{1374,460116,880026},2},</div>
<div>                          {{1374,460116,880026},2},</div><div>                          0.0016008717049531846,0.0016008717049531846,</div><div>                          {1374,461800,598037}}},</div><div>                        0,</div>
<div>                        {dict,0,16,16,8,80,48,</div><div>                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div>                        [],</div><div>                        {dict,0,16,16,8,80,48,</div><div>                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div>                        {set,0,16,16,8,80,48,</div><div>                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},</div>
<div>                       {[],[]},</div><div>                       {[],[]},</div><div>                       undefined,undefined,#Ref&lt;0.0.742.237570&gt;,undefined,</div><div>                       {state,none,5000,undefined},</div>
<div>                       {dict,0,16,16,8,80,48,</div><div>                        {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                        {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</div>
<div>                       undefined,undefined},</div><div>                      {noproc,</div><div>                       {gen_server2,call,</div><div>                        [msg_store_transient,</div><div>                         {client_terminate,&lt;&lt;&quot;ܺ�o$��W����\b?J&quot;&gt;&gt;},</div>
<div>                         infinity]}},</div><div>                      shutdown]}}</div><div>** When handler state == {resource,&lt;&lt;&quot;/&quot;&gt;&gt;,exchange,&lt;&lt;&quot;amq.rabbitmq.log&quot;&gt;&gt;}</div>
<div>** Reason == {badarg,[{ets,lookup,[rabbit_registry,{exchange,topic}]},</div><div>                      {rabbit_registry,lookup_module,2},</div><div>                      {rabbit_exchange,type_to_module,1},</div><div>
                      {rabbit_exchange,route1,2},</div><div>                      {rabbit_basic,publish,2},</div><div>                      {rabbit_error_logger,publish1,4},</div><div>                      {rabbit_error_logger,handle_event,2},</div>
<div>                      {gen_event,server_update,4}]}</div><div><br></div><div>=INFO REPORT==== 22-Jul-2013::14:50:27 ===</div><div>    application: rabbit</div><div>    exited: shutdown</div><div>    type: permanent</div>
</div><div><br></div></div>