[rabbitmq-discuss] rabbitmq 2.4.1 failure

Tue Apr 17 18:03:52 BST 2012

We recently had a rabbitmq broker fail and are unable to determine the
root cause. The broker is version 2.4.1, erlang is 14B02, running on
ubuntu 10.04 LTS.

SASL log snippet

=SUPERVISOR REPORT==== 11-Apr-2012::00:15:30 ===
     Supervisor: {<0.5419.34>,rabbit_channel_sup_sup}
     Context:    shutdown_error
     Reason:     shutdown
     Offender:   [{pid,<0.5731.34>},
                  {name,channel_sup},
                  {mfa,{rabbit_channel_sup,start_link,[]}},
                  {restart_type,temporary},
                  {shutdown,infinity},
                  {child_type,supervisor}]

=CRASH REPORT==== 11-Apr-2012::05:04:32 ===
  crasher:
    initial call: gen:init_it/7
    pid: <0.232.0>
    registered_name: msg_store_transient
    exception exit: {{badmatch,false},
                     [{rabbit_msg_store_ets_index,insert,2},
                      {rabbit_msg_store,write_message,3},
                      {rabbit_msg_store,handle_cast,2},
                      {gen_server2,handle_msg,2},
                      {proc_lib,wake_up,3}]}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_sup,<0.147.0>]
    messages: [{'EXIT',<0.233.0>,normal}]
    links: [<0.148.0>]
    dictionary: [{fhc_age_tree,{0,nil}}]
    trap_exit: true
    status: running
    heap_size: 10946
    stack_size: 24
    reductions: 98380626
  neighbours:

RABBIMQ log snippet:

=INFO REPORT==== 11-Apr-2012::05:04:31 ===
closing TCP connection <0.10243.65> from 10.70.40.77:53173

=ERROR REPORT==== 11-Apr-2012::05:04:31 ===
** Generic server msg_store_transient terminating
** Last message in was {'$gen_cast',
                           {client_dying,

<<74,18,61,37,8,55,8,91,210,27,70,185,112,89,
                                 171,154>>}}
** When Server state == {msstate,
                         "/var/lib/rabbitmq/mnesia/rabbit at che-
csebrokerp1/msg_store_transient",
                         rabbit_msg_store_ets_index,
                         {state,417861,
                          "/var/lib/rabbitmq/mnesia/rabbit at che-
csebrokerp1/msg_store_transient"},
                         0,#Ref<0.0.0.875>,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
                         [],undefined,0,12073198,[],<0.233.0>,
421958,413764,
                         426055,
....skipping...
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
                         undefined,undefined}
** Reason for termination ==
** {noproc,
       {gen_server2,call,
           [msg_store_transient,
            {client_terminate,

<<17,102,9,148,6,184,165,141,162,246,194,57,36,62,208,135>>},
            infinity]}}
** In 'terminate' callback with reason ==
** shutdown

=ERROR REPORT==== 11-Apr-2012::05:04:43 ===
** gen_event handler rabbit_error_logger crashed.
** Was installed in error_logger
** Last event was: {error,<0.146.0>,
                    {<0.9700.6>,
                     "** Generic server ~p terminating~n** Last
message in was ~p~n** When Server state == ~p~n** Reason for
termination == ~n** ~p~n** In 'terminate' callback with reason ==~n**
~p~n",
                     [<0.9700.6>,
                      {'EXIT',<0.241.0>,shutdown},
                      {q,
                       {amqqueue,
                        {resource,<<"/rssm">>,queue,
                         <<"cse.rssm.logManager.sqlserver">>},
...more...

After the failure we were not able to restart rabbitmq, it required a
system reboot. On the attempted restart we saw the message regarding
activating plugins, but hung and never completed. We don't have much
more information except the logs, which we are not able to fully
understand.

Thanks,
Mark.