[rabbitmq-discuss] RabbitMQ crashes

[Matti Linnanvuori]

RabbitMQ no longer responds to queries and /etc/init.d/rabbitmq-server
stop hung. RabbitMQ version is 2.5.1. SUSE Linux Enterprise Server 11
SP1.

uname -a
Linux iac-pgis 2.6.32.36-0.5-default #1 SMP 2011-04-14 10:12:31 +0200
x86_64 x86_64 x86_64 GNU/Linux

/etc/init.d/rabbitmq-server status

Status of node 'rabbit at iac-pgis' ...
[{pid,2770},
 {running_applications,[{os_mon,"CPO  CXC 138 46","2.2.6"},
                        {sasl,"SASL  CXC 138 11","2.1.9.4"},
                        {mnesia,"MNESIA  CXC 138 12","4.4.19"},
                        {stdlib,"ERTS  CXC 138 10","1.17.4"},
                        {kernel,"ERTS  CXC 138 10","2.14.4"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R14B03 (erts-5.8.4) [source] [64-bit] [rq:1]
[async-threads:30] [hipe] [kernel-poll:true]\n"},
 {memory,[{total,96861600},
          {processes,10802024},
          {processes_used,10792904},
          {system,86059576},
          {atom,1113929},
          {atom_used,1104932},
          {binary,734480},
          {code,11023444},
          {ets,71867888}]}]
...done.

rabbit at iac-pgis-sasl.log is the following:
=CRASH REPORT==== 29-Aug-2011::14:27:43 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.118.0>
    registered_name: []
    exception exit: {{badmatch,eof},
                     [{rabbit_queue_index,load_segment,2},
                      {rabbit_queue_index,segment_entries_foldr,3},
                      {rabbit_queue_index,
                          '-queue_index_walker_reader/2-lc$^0/1-0-',
4},
                      {rabbit_queue_index,queue_index_walker_reader,
2},
                      {worker_pool_worker,handle_cast,2},
                      {gen_server2,handle_msg,2},
                      {proc_lib,init_p_do_apply,3}]}
      in function  gen_server2:terminate/3
    ancestors: [worker_pool_sup,rabbit_sup,<0.106.0>]
    messages: []
    links: [#Port<0.2307>,<0.115.0>,#Port<0.2274>]
    dictionary: [{{#Ref<0.0.0.35370>,fhc_handle},
                   {handle,{file_descriptor,prim_file,{#Port<0.2307>,
18}},
                           0,0,false,0,0,[],true,
                           "/var/lib/rabbitmq/mnesia/rabbit at iac-pgis/
queues/66AS0RWVNJTRSJ1ZWY9DKL71X/139.idx",
                           [{read_ahead,491520},binary,raw,read],
                           [],false,true,
                           {1314,624463,414214}}},
                  {{"/var/lib/rabbitmq/mnesia/rabbit at iac-pgis/queues/
66AS0RWVNJTRSJ1ZWY9DKL71X/journal.jif",
                    fhc_file},
                   {file,1,true}},
                  {worker_pool_worker,true},
                  {{"/var/lib/rabbitmq/mnesia/rabbit at iac-pgis/queues/
66AS0RWVNJTRSJ1ZWY9DKL71X/139.idx",
                    fhc_file},
                   {file,1,false}},
                  {fhc_age_tree,{2,
                                 {{1314,624458,952622},
                                  #Ref<0.0.0.7784>,nil,
                                  {{1314,624463,414214},
                                   #Ref<0.0.0.35370>,nil,nil}}}},
                  {{#Ref<0.0.0.7784>,fhc_handle},
                   {handle,{file_descriptor,prim_file,{#Port<0.2274>,
17}},
                           0,0,false,0,infinity,[],true,
                           "/var/lib/rabbitmq/mnesia/rabbit at iac-pgis/
queues/66AS0RWVNJTRSJ1ZWY9DKL71X/journal.jif",
                           [write,binary,raw,read],
                           [{write_buffer,infinity}],
                           true,true,
                           {1314,624458,952622}}}]
    trap_exit: false
    status: running
    heap_size: 514229
    stack_size: 24
    reductions: 18980671
  neighbours:

=SUPERVISOR REPORT==== 29-Aug-2011::14:27:43 ===
     Supervisor: {local,worker_pool_sup}
     Context:    child_terminated
     Reason:     {{badmatch,eof},
                  [{rabbit_queue_index,load_segment,2},
                   {rabbit_queue_index,segment_entries_foldr,3},
                   {rabbit_queue_index,
                       '-queue_index_walker_reader/2-lc$^0/1-0-',4},
                   {rabbit_queue_index,queue_index_walker_reader,2},
                   {worker_pool_worker,handle_cast,2},
                   {gen_server2,handle_msg,2},
                   {proc_lib,init_p_do_apply,3}]}
     Offender:   [{pid,<0.118.0>},
                  {name,1},
                  {mfargs,{worker_pool_worker,start_link,[1]}},
                  {restart_type,transient},
                  {shutdown,4294967295},
                  {child_type,worker}]

rabbit at iac-pgis.log is the following:

=INFO REPORT==== 29-Aug-2011::14:27:37 ===
Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 29-Aug-2011::14:27:37 ===
Memory limit set to 210MB.

=INFO REPORT==== 29-Aug-2011::14:27:37 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 29-Aug-2011::14:27:37 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide
index

=WARNING REPORT==== 29-Aug-2011::14:27:37 ===
msg_store_persistent: rebuilding indices from scratch

=ERROR REPORT==== 29-Aug-2011::14:27:43 ===
** Generic server <0.118.0> terminating
** Last message in was {'$gen_cast',
                           {submit_async,
                               #Fun<rabbit_queue_index.24.55115249>}}
** When Server state == 1
** Reason for termination ==
** {{badmatch,eof},
    [{rabbit_queue_index,load_segment,2},
     {rabbit_queue_index,segment_entries_foldr,3},
     {rabbit_queue_index,'-queue_index_walker_reader/2-lc$^0/1-0-',4},
     {rabbit_queue_index,queue_index_walker_reader,2},
     {worker_pool_worker,handle_cast,2},
     {gen_server2,handle_msg,2},
     {proc_lib,init_p_do_apply,3}]}