[rabbitmq-discuss] RabbitMQ stopping when trying to delete locked mnesia files?

Øyvind Tjervaag oyvind at tjervaag.com
Tue May 29 13:01:48 BST 2012


Hi!

I've had this problem a couple of times now where it seems, if I'm deciphering the logs correctly, the RabbitMQ crashed when it tries to delete mnesia files that are locked by backup or virus-scanning. I thought I had seen something about this on this list a while back, but I can't seem to find it now.

Sorry about the long logs, but I've snipped what I thought was irrelevant:


=ERROR REPORT==== 24-May-2012::01:20:54 ===
** Generic server <0.195.0> terminating
** Last message in was {'$gen_cast',{delete,8611}}
** When Server state == {state,
                            {dict,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            [],
                            {gc_state,
                                "e:/Mnesia/rabbit at machinename-mnesia/msg_store_persistent",
                                rabbit_msg_store_ets_index,
                                {state,213063,
                                    "e:/Mnesia/rabbit at machinename-mnesia/msg_store_persistent"},
                                208966,217160,<0.192.0>}}
** Reason for termination == 
** {{badmatch,{error,eacces}},
    [{rabbit_msg_store,safe_file_delete,3,[]},
     {rabbit_msg_store_gc,'-attempt_action/3-fun-0-',1,[]},
     {lists,'-filter/2-lc$^0/1-0-',2,[{file,"lists.erl"},{line,1220}]},
     {rabbit_msg_store_gc,attempt_action,3,[]},
     {rabbit_msg_store_gc,handle_cast,2,[]},
     {gen_server2,handle_msg,2,[]},
     {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}

=ERROR REPORT==== 24-May-2012::01:20:54 ===
** Generic server msg_store_persistent terminating
** Last message in was {'EXIT',<0.195.0>,
                           {{badmatch,{error,eacces}},
                            [{rabbit_msg_store,safe_file_delete,3,[]},
                             {rabbit_msg_store_gc,'-attempt_action/3-fun-0-',
                                 1,[]},
                             {lists,'-filter/2-lc$^0/1-0-',2,
                                 [{file,"lists.erl"},{line,1220}]},
                             {rabbit_msg_store_gc,attempt_action,3,[]},
                             {rabbit_msg_store_gc,handle_cast,2,[]},
                             {gen_server2,handle_msg,2,[]},
                             {proc_lib,wake_up,3,
                                 [{file,"proc_lib.erl"},{line,237}]}]}}
** When Server state == {msstate,
                            "e:/Mnesia/rabbit at machinename-mnesia/msg_store_persistent",
                            rabbit_msg_store_ets_index,
                            {state,213063,
                                "e:/Mnesia/rabbit at machinename-mnesia/msg_store_persistent"},
                            8612,#Ref<0.0.100.15376>,
                            {dict,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            undefined,6556,6556,[],<0.195.0>,217160,208966,
                            221257,225354,
                            {set,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            {dict,3,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],
                                  [[<<242,131,46,148,249,165,117,168,89,192,
                                      116,172,97,9,69,105>>|
                                    {<0.201.0>,
                                     #Fun<rabbit_variable_queue.1.12380193>,
                                     #Fun<rabbit_variable_queue.13.91199175>}]],
                                  [],
                                  [[<<93,218,27,224,115,59,29,211,84,246,239,
                                      132,234,163,75,58>>|
                                    {<0.200.0>,
                                     #Fun<rabbit_variable_queue.1.12380193>,
                                     #Fun<rabbit_variable_queue.13.91199175>}]],
                                  [[<<11,218,33,254,192,34,57,180,178,129,31,
                                      72,69,118,184,70>>|
                                    {<0.199.0>,
                                     #Fun<rabbit_variable_queue.1.12380193>,
                                     #Fun<rabbit_variable_queue.13.91199175>}]],
                                  [],[],[],[],[],[],[],[],[],[],[]}}},
                            false,16777216,
                            {dict,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}}}
** Reason for termination == 
** {noproc,{gen_server2,call,[<0.195.0>,stop,infinity]}}
** In 'terminate' callback with reason ==
** {{badmatch,{error,eacces}},
    [{rabbit_msg_store,safe_file_delete,3,[]},
     {rabbit_msg_store_gc,'-attempt_action/3-fun-0-',1,[]},
     {lists,'-filter/2-lc$^0/1-0-',2,[{file,"lists.erl"},{line,1220}]},
     {rabbit_msg_store_gc,attempt_action,3,[]},
     {rabbit_msg_store_gc,handle_cast,2,[]},
     {gen_server2,handle_msg,2,[]},
     {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}

=INFO REPORT==== 24-May-2012::01:20:54 ===
stopped TCP Listener on 0.0.0.0:5672

=INFO REPORT==== 24-May-2012::01:20:54 ===
stopped TCP Listener on [::]:5672

=ERROR REPORT==== 24-May-2012::01:21:01 ===
** Generic server <0.200.0> terminating
** Last message in was {'EXIT',<0.198.0>,shutdown}
** When Server state == {q,
                         {amqqueue,
                          {resource,<<"/">>,queue,<<"QUEUENAME">>},
                          true,false,none,[],<0.200.0>,[],undefined},
                         none,true,rabbit_variable_queue,
                         {vqstate,
                          {0,{[],[]}},
                          {0,{[],[]}},
                          {delta,undefined,0,undefined},
                          {0,{[],[]}},
                          {0,{[],[]}},
                          793942,
                          {0,nil},
                          undefined,
                          {0,nil},
                          {qistate,
                           "e:/Mnesia/rabbit at machinename-mnesia/queues/CDBIXPAYNE7Z1MBHQJSYJB9T9",
                           {{dict,4,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],
                               [[44|
                                 {segment,44,
                                  "e:/Mnesia/rabbit at machinename-mnesia/queues/CDBIXPAYNE7Z1MBHQJSYJB9T9/44.idx",
                                  {array,16384,0,undefined,
                                   {{{{{{{<<177,189,215,156,204,89,5,250,132,
                                            27,178,87,129,157,90,185>>,
                                          {message_properties,undefined,true},
                                          true},
                                         del,ack},
                                        {{<<219,139,40,233,147,201,159,68,150,
                                            79,119,170,23,210,45,19>>,
                                          {message_properties,undefined,true},
                                          true},
                                         del,ack},
<---------- BIG SNIP ------------------>
                            {state,192578,
                             "e:/Mnesia/rabbit at machinename-mnesia/msg_store_transient"},
                            rabbit_msg_store_ets_index,
                            "e:/Mnesia/rabbit at machinename-mnesia/msg_store_transient",
                            <0.190.0>,196675,188474,200772,204869}},
                          true,0,#Fun<rabbit_amqqueue_process.5.17325602>,2,2,
                          infinity,2,0,0,0,2,
                          {rates,
                           {{1337,815248,910691},10},
                           {{1337,815248,910691},10},
                           1.4544749345486279,1.4544749345486279,
                           {1337,815253,957691}},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          0,0,
                          {rates,
                           {{1337,815248,910691},10},
                           {{1337,815248,910691},10},
                           1.4544749345486279,1.4544749345486279,
                           {1337,815253,957691}}},
                         {[],[]},
                         undefined,undefined,#Ref<0.0.100.15371>,undefined,
                         {state,fine,5000,#Ref<0.0.100.15372>},
                         {0,nil},
                         undefined,undefined,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         1,
                         {{0,nil},{0,nil}},
                         undefined,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         undefined,undefined}
** Reason for termination == 
** {noproc,
       {gen_server2,call,
           [msg_store_persistent,
            {client_terminate,
                <<242,131,46,148,249,165,117,168,89,192,116,172,97,9,69,105>>},
            infinity]}}
** In 'terminate' callback with reason ==
** shutdown

=INFO REPORT==== 24-May-2012::01:21:43 ===
    application: rabbit
    exited: shutdown
    type: permanent


I'm running RabbitMQ version 2.8.2 and Erlang R15B01 on Windows 2008 R2 (64 bit). Now, I've told the IT-ops-people to stop locking any files in the mnesia folder, but I think RabbitMQ should not fail this badly when it tries deleting a locked file? 

Or am I doing something else wrong here?

thanks,
Øyvind


More information about the rabbitmq-discuss mailing list