[rabbitmq-discuss] Cluster Node Crashes

Cordell, Ron Ron.Cordell at RelayHealth.com
Tue Sep 17 23:48:24 BST 2013


We are running RabbitMQ 3.1.5 and Erlang 16B01 x64 on Windows Server 2008R2 in a clustered configuration.

We are seeing nodes crash at random and don't understand what's going on. However, there is a common theme in the logs, of which a sample is here:

Does anyone have any insight into what might be happening? We see this in several different environments with similar cluster configurations and all running the same RabbitMQ/Erlang versions…

=INFO REPORT==== 13-Sep-2013::15:10:10 ===
accepting AMQP connection <0.5121.11> (10.100.183.10:63072 -> 10.100.183.75:5672)

=INFO REPORT==== 13-Sep-2013::15:10:10 ===
accepting AMQP connection <0.5149.11> (10.100.183.10:63075 -> 10.100.183.75:5672)

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_sendtopracticefusionactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.416.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.409.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-message_tracking_control_sjpeweb01' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.532.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.529.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_lookupproviderinregistryactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.352.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.349.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-compensate_sendtocloverleafactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.356.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.357.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-subscriber_activity_service_control_sjpeweb02' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.376.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.373.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_fakefaultedactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.348.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.345.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-inbound_activity_service_control_sjpeweb04' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.388.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.385.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_mongopersisteractivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.320.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.317.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-compensate_sendtorouteractivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.344.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.341.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-compensate_preparedocumentforroutingactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.384.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.381.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_unrecognizedmessageactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.316.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.313.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_sendtocloverleafactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.360.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.353.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-inbound_activity_service_control_sjpeweb02' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.372.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.369.0>

=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-exchange_infrastructure_control_sjpeweb02' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.304.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.301.0>

=ERROR REPORT==== 15-Sep-2013::06:34:16 ===
** Generic server <0.336.0> terminating
** Last message in was {gm_deaths,[<5111.330.0>]}
** When Server state == {state,
                         {amqqueue,
                          {resource,<<"vortex">>,queue,
                           <<"ha-execute_sendtorouteractivity">>},
                          true,false,none,
                          [{<<"x-ha-policy">>,longstr,<<"all">>}],
                          <5111.329.0>,
                          [<5111.329.0>,<7010.329.0>,<7011.333.0>],
                          [<7011.333.0>,<7010.329.0>,<5111.329.0>],
                          [{vhost,<<"vortex">>},
                           {name,<<"ha-policy">>},
                           {pattern,<<"^ha-.*">>},
                           {definition,
                            [{<<"ha-mode">>,<<"all">>},
                             {<<"ha-sync-mode">>,<<"automatic">>}]},
                           {priority,0}],
                          [{<7011.334.0>,<7011.333.0>},
                           {<7010.330.0>,<7010.329.0>},
                           {<5111.330.0>,<5111.329.0>},
                           {<7009.328.0>,<7009.327.0>}]},
                         <0.337.0>,rabbit_variable_queue,
                         {vqstate,
                          {0,{[],[]}},
                          {0,{[],[]}},
                          {delta,undefined,0,undefined},
                          {0,{[],[]}},
                          {0,{[],[]}},
                          0,
                          {0,nil},
                          {0,nil},
                          {qistate,
                           "e:/RabbitMQ/db/rabbit-mnesia/queues/CZDXJN5ZJBZN98DWOHU2MZHL",
                           {{dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            []},
                           undefined,0,65536,
                           #Fun<rabbit_variable_queue.2.81334491>,
                           {0,nil}},
                          {{client_msstate,msg_store_persistent,
                            <<127,33,92,106,192,40,110,10,76,157,95,226,1,93,
                              82,226>>,
                            {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,348236,
                             "e:/RabbitMQ/db/rabbit-mnesia/msg_store_persistent"},
                            rabbit_msg_store_ets_index,
                            "e:/RabbitMQ/db/rabbit-mnesia/msg_store_persistent",
                            <0.269.0>,352333,344139,356430,360527},
                           {client_msstate,msg_store_transient,
                            <<95,20,53,215,211,161,4,63,158,97,9,63,159,60,91,
                              221>>,
                            {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,327751,
                             "e:/RabbitMQ/db/rabbit-mnesia/msg_store_transient"},
                            rabbit_msg_store_ets_index,
                            "e:/RabbitMQ/db/rabbit-mnesia/msg_store_transient",
                            <0.264.0>,331848,323654,335945,340042}},
                          true,0,0,0,infinity,0,0,0,0,0,
                          {rates,
                           {{1379,109357,6220},0},
                           {{1379,109357,6220},0},
                           0.0,0.0,
                           {1379,109829,181271}},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          0,0,
                          {rates,
                           {{1379,109357,6220},0},
                           {{1379,109357,6220},0},
                           0.0,0.0,
                           {1379,109829,181271}}},
                         undefined,#Ref<0.0.55.58770>,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         0}
** Reason for termination ==
** {{badmatch,{error,not_found}},
    [{rabbit_amqqueue_process,i,2,[]},
     {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
     {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
     {rabbit_amqqueue_process,emit_stats,2,[]},
     {rabbit_event,if_enabled,3,[]},
     {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},
     {rabbit_amqqueue_process,terminate_shutdown,2,[]},
     {gen_server2,terminate,3,[]}]}
** In 'terminate' callback with reason ==
** normal

=INFO REPORT==== 15-Sep-2013::06:34:16 ===
closing AMQP connection <0.4955.11> (10.100.183.10:62972 -> 10.100.183.75:5672)

=INFO REPORT==== 15-Sep-2013::06:34:16 ===
closing AMQP connection <0.3177.11> (10.100.183.10:49728 -> 10.100.183.75:5672)

=INFO REPORT==== 15-Sep-2013::06:34:25 ===
accepting AMQP connection <0.24239.16> (10.100.183.10:56699 -> 10.100.183.75:5672)

=INFO REPORT==== 15-Sep-2013::06:34:25 ===
accepting AMQP connection <0.24248.16> (10.100.183.10:55592 -> 10


And a little later in the same log, here:

=INFO REPORT==== 15-Sep-2013::06:34:25 ===
Synchronising queue 'ha-compensate_fakecompensationfaultedactivity' in vhost 'vortex': all slaves already synced

=INFO REPORT==== 15-Sep-2013::06:34:25 ===
Synchronising queue 'ha-compensate_fakecompensationfaultedactivity' in vhost 'vortex': 0 messages to synchronise

=INFO REPORT==== 15-Sep-2013::06:34:25 ===
Synchronising queue 'ha-compensate_fakecompensationfaultedactivity' in vhost 'vortex': all slaves already synced

=INFO REPORT==== 15-Sep-2013::06:34:25 ===
Synchronising queue 'ha-compensate_fakecompensationfaultedactivity' in vhost 'vortex': 0 messages to synchronise

=INFO REPORT==== 15-Sep-2013::06:34:25 ===
Synchronising queue 'ha-compensate_fakecompensationfaultedactivity' in vhost 'vortex': all slaves already synced

=INFO REPORT==== 15-Sep-2013::06:34:45 ===
accepting AMQP connection <0.24463.16> (10.100.183.10:55948 -> 10.100.183.75:5672)

=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
** Generic server <0.353.0> terminating
** Last message in was {'DOWN',#Ref<0.0.0.3812>,process,<5111.350.0>,
                           {function_clause,
                               [{orddict,fetch,
                                    [{9,<5111.350.0>},[]],
                                    [{file,"orddict.erl"},{line,72}]},
                                {gm,check_neighbours,1,[]},
                                {gm,handle_cast,2,[]},
                                {gen_server2,handle_msg,2,[]},
                                {proc_lib,wake_up,3,
                                    [{file,"proc_lib.erl"},{line,249}]}]}}
** When Server state == {state,
                            {18,<0.353.0>},
                            {{22,<0.24486.16>},#Ref<0.0.55.61004>},
                            {{9,<5111.350.0>},#Ref<0.0.0.3812>},
                            {resource,<<"vortex">>,queue,
                                <<"ha-execute_lookupproviderinregistryactivity">>},
                            rabbit_mirror_queue_slave,
                            {24,
                             [{{9,<5111.350.0>},
                               {view_member,
                                   {9,<5111.350.0>},
                                   [],
                                   {18,<0.353.0>},
                                   {12,<7010.350.0>}}},
                              {{12,<7010.350.0>},
                               {view_member,
                                   {12,<7010.350.0>},
                                   [],
                                   {9,<5111.350.0>},
                                   {22,<0.24486.16>}}},
                              {{18,<0.353.0>},
                               {view_member,
                                   {18,<0.353.0>},
                                   [],
                                   {22,<0.24486.16>},
                                   {9,<5111.350.0>}}},
                              {{22,<0.24486.16>},
                               {view_member,
                                   {22,<0.24486.16>},
                                   [],
                                   {12,<7010.350.0>},
                                   {18,<0.353.0>}}}]},
                            0,
                            [{{9,<5111.350.0>},
                              {member,{[{2,process_death}],[]},2,1}},
                             {{12,<7010.350.0>},
                              {member,{[{2,process_death}],[]},2,1}},
                             {{18,<0.353.0>},{member,{[],[]},0,0}},
                             {{22,<0.24486.16>},{member,{[],[]},-1,-1}}],
                            [<0.352.0>],
                            {[],[]},
                            [],undefined,
                            #Fun<rabbit_misc.execute_mnesia_transaction.1>}
** Reason for termination ==
** {function_clause,[{orddict,fetch,
                              [{18,<0.353.0>},[]],
                              [{file,"orddict.erl"},{line,72}]},
                     {gm,check_neighbours,1,[]},
                     {gm,handle_info,2,[]},
                     {gen_server2,handle_msg,2,[]},
                     {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}

=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
** Generic server <0.24486.16> terminating
** Last message in was {'DOWN',#Ref<0.0.55.60998>,process,<0.353.0>,
                           {function_clause,
                               [{orddict,fetch,
                                    [{18,<0.353.0>},[]],
                                    [{file,"orddict.erl"},{line,72}]},
                                {gm,check_neighbours,1,[]},
                                {gm,handle_info,2,[]},
                                {gen_server2,handle_msg,2,[]},
                                {proc_lib,wake_up,3,
                                    [{file,"proc_lib.erl"},{line,249}]}]}}
** When Server state == {state,
                            {22,<0.24486.16>},
                            {{12,<7010.350.0>},#Ref<0.0.55.60997>},
                            {{18,<0.353.0>},#Ref<0.0.55.60998>},
                            {resource,<<"vortex">>,queue,
                                <<"ha-execute_lookupproviderinregistryactivity">>},
                            rabbit_mirror_queue_coordinator,
                            {23,
                             [{{9,<5111.350.0>},
                               {view_member,
                                   {9,<5111.350.0>},
                                   [],
                                   {18,<0.353.0>},
                                   {12,<7010.350.0>}}},
                              {{12,<7010.350.0>},
                               {view_member,
                                   {12,<7010.350.0>},
                                   [],
                                   {9,<5111.350.0>},
                                   {22,<0.24486.16>}}},
                              {{18,<0.353.0>},
                               {view_member,
                                   {18,<0.353.0>},
                                   [{6,<7009.348.0>}],
                                   {22,<0.24486.16>},
                                   {9,<5111.350.0>}}},
                              {{22,<0.24486.16>},
                               {view_member,
                                   {22,<0.24486.16>},
                                   [],
                                   {12,<7010.350.0>},
                                   {18,<0.353.0>}}}]},
                            -1,
                            [{{6,<7009.348.0>},{member,{[],[]},15,15}},
                             {{9,<5111.350.0>},
                              {member,{[{2,process_death}],[]},2,1}},
                             {{12,<7010.350.0>},
                              {member,{[{2,process_death}],[]},2,1}},
                             {{18,<0.353.0>},{member,{[],[]},0,0}},
                             {{22,<0.24486.16>},{member,{[],[]},-1,-1}}],
                            [<0.24485.16>],
                            {[],[]},
                            [],undefined,
                            #Fun<rabbit_misc.execute_mnesia_transaction.1>}
** Reason for termination ==
** {function_clause,[{orddict,fetch,
                              [{22,<0.24486.16>},[]],
                              [{file,"orddict.erl"},{line,72}]},
                     {gm,check_neighbours,1,[]},
                     {gm,handle_info,2,[]},
                     {gen_server2,handle_msg,2,[]},
                     {proc_lib,init_p_do_apply,3,
                               [{file,"proc_lib.erl"},{line,239}]}]}

=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
** Generic server <0.352.0> terminating
** Last message in was {gm_deaths,[<0.24486.16>,<0.353.0>,<7010.350.0>,
                                   <5111.350.0>]}
** When Server state == {state,
                         {amqqueue,
                          {resource,<<"vortex">>,queue,
                           <<"ha-execute_lookupproviderinregistryactivity">>},
                          true,false,none,
                          [{<<"x-ha-policy">>,longstr,<<"all">>}],
                          <0.24484.16>,
                          [<5111.349.0>,<7010.349.0>,<7011.349.0>],
                          [<7011.349.0>,<7010.349.0>,<5111.349.0>],
                          [{vhost,<<"vortex">>},
                           {name,<<"ha-policy">>},
                           {pattern,<<"^ha-.*">>},
                           {definition,
                            [{<<"ha-mode">>,<<"all">>},
                             {<<"ha-sync-mode">>,<<"automatic">>}]},
                           {priority,0}],
                          [{<7011.350.0>,<7011.349.0>},
                           {<7010.350.0>,<7010.349.0>},
                           {<5111.350.0>,<5111.349.0>},
                           {<7009.348.0>,<7009.347.0>}]},
                         <0.353.0>,rabbit_variable_queue,
                         {vqstate,
                          {0,{[],[]}},
                          {0,{[],[]}},
                          {delta,undefined,0,undefined},
                          {0,{[],[]}},
                          {0,{[],[]}},
                          0,
                          {0,nil},
                          {0,nil},
                          {qistate,
                           "e:/RabbitMQ/db/rabbit-mnesia/queues/5MD5HQ7PCUA3C2UP3Z9HUANOV",
                           {{dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            []},
                           undefined,0,65536,
                           #Fun<rabbit_variable_queue.2.81334491>,
                           {0,nil}},
                          {{client_msstate,msg_store_persistent,
                            <<139,43,92,149,89,56,222,175,208,46,119,138,91,
                              209,199,19>>,
                            {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,348236,
                             "e:/RabbitMQ/db/rabbit-mnesia/msg_store_persistent"},
                            rabbit_msg_store_ets_index,
                            "e:/RabbitMQ/db/rabbit-mnesia/msg_store_persistent",
                            <0.269.0>,352333,344139,356430,360527},
                           {client_msstate,msg_store_transient,
                            <<241,205,183,204,120,219,30,233,243,36,174,112,
                              168,245,105,99>>,
                            {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,327751,
                             "e:/RabbitMQ/db/rabbit-mnesia/msg_store_transient"},
                            rabbit_msg_store_ets_index,
                            "e:/RabbitMQ/db/rabbit-mnesia/msg_store_transient",
                            <0.264.0>,331848,323654,335945,340042}},
                          true,0,0,0,infinity,0,0,0,0,0,
                          {rates,
                           {{1379,109827,41270},0},
                           {{1379,109827,41270},0},
                           0.0,0.0,
                           {1379,252056,729281}},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          0,0,
                          {rates,
                           {{1379,109827,41270},0},
                           {{1379,109827,41270},0},
                           0.0,0.0,
                           {1379,252056,729281}}},
                         undefined,#Ref<0.0.55.61023>,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         0}
** Reason for termination ==
** {{badmatch,{error,not_found}},
    [{rabbit_amqqueue_process,i,2,[]},
     {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
     {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
     {rabbit_amqqueue_process,emit_stats,2,[]},
     {rabbit_event,if_enabled,3,[]},
     {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},
     {rabbit_amqqueue_process,terminate_shutdown,2,[]},
     {gen_server2,terminate,3,[]}]}
** In 'terminate' callback with reason ==
** normal

=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
** Generic server <0.24484.16> terminating
** Last message in was {init,new}
** When Server state == {q,{amqqueue,
                               {resource,<<"vortex">>,queue,
                                   <<"ha-execute_lookupproviderinregistryactivity">>},
                               true,false,none,
                               [{<<"x-ha-policy">>,longstr,<<"all">>}],
                               <0.24484.16>,[],[],
                               [{vhost,<<"vortex">>},
                                {name,<<"ha-policy">>},
                                {pattern,<<"^ha-.*">>},
                                {definition,
                                    [{<<"ha-mode">>,<<"all">>},
                                     {<<"ha-sync-mode">>,<<"automatic">>}]},
                                {priority,0}],
                               []},
                           none,false,undefined,undefined,
                           {[],[]},
                           undefined,undefined,undefined,undefined,
                           {state,fine,5000,undefined},
                           {0,nil},
                           undefined,undefined,undefined,
                           {dict,0,16,16,8,80,48,
                               {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []},
                               {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []}}},
                           undefined,undefined,undefined,running}
** Reason for termination ==
** {{badmatch,{error,not_found}},
    [{rabbit_amqqueue_process,i,2,[]},
     {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
     {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
     {rabbit_amqqueue_process,emit_stats,2,[]},
     {rabbit_event,if_enabled,3,[]},
     {rabbit_amqqueue_process,declare,3,[]},
     {gen_server2,handle_msg,2,[]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
** Generic server <0.24482.16> terminating
** Last message in was {'$gen_cast',
                           {method,
                               {'basic.consume',0,
                                   <<"ha-execute_lookupproviderinregistryactivity">>,
                                   <<>>,false,false,false,false,[]},
                               none,noflow}}
** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,2,<0.24463.16>,
                            <0.24480.16>,<0.24463.16>,
                            <<"10.100.183.10:55948 -> 10.100.183.75:5672">>,
                            {lstate,<0.24481.16>,true,false},
                            none,1,
                            {[],[]},
                            {user,<<"SJPERMQVORTEXWRITE">>,[],
                                rabbit_auth_backend_ldap,
                                {impl,
                                    "CN=SJPERMQ VORTEXWRITE,OU=PERMQ,OU=PERFORMANCE,OU=Environments,OU=Servers,DC=RHB,DC=AD",
                                    <<"5k!Kz_T86Xa.W~">>}},
                            <<"vortex">>,
                            <<"ha-execute_lookupproviderinregistryactivity">>,
                            {dict,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            {dict,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            {dict,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            {set,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            {dict,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            {set,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                            <0.24461.16>,
                            {state,fine,5000,#Ref<0.0.55.60968>},
                            false,1,
                            {{0,nil},{0,nil}},
                            [],
                            [{<<"publisher_confirms">>,bool,true},
                             {<<"exchange_exchange_bindings">>,bool,true},
                             {<<"consumer_cancel_notify">>,bool,true},
                             {<<"basic.nack">>,bool,true}],
                            none}
** Reason for termination ==
** {{{{badmatch,{error,not_found}},
      [{rabbit_amqqueue_process,i,2,[]},
       {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
       {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
       {rabbit_amqqueue_process,emit_stats,2,[]},
       {rabbit_event,if_enabled,3,[]},
       {rabbit_amqqueue_process,declare,3,[]},
       {gen_server2,handle_msg,2,[]},
       {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},
     {gen_server2,call,
         [<0.24484.16>,
          {basic_consume,false,<0.24482.16>,<0.24481.16>,true,
              <<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>,false,none,
              {'basic.consume_ok',<<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>}},
          infinity]}},
    [{gen_server2,call,3,[]},
     {rabbit_channel,'-handle_method/3-fun-1-',7,[]},
     {rabbit_misc,with_exit_handler,2,[]},
     {rabbit_channel,handle_method,3,[]},
     {rabbit_channel,handle_cast,2,[]},
     {gen_server2,handle_msg,2,[]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
AMQP connection <0.24463.16> (running), channel 2 - error:
{{{{badmatch,{error,not_found}},
   [{rabbit_amqqueue_process,i,2,[]},
    {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
    {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
    {rabbit_amqqueue_process,emit_stats,2,[]},
    {rabbit_event,if_enabled,3,[]},
    {rabbit_amqqueue_process,declare,3,[]},
    {gen_server2,handle_msg,2,[]},
    {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},
  {gen_server2,call,
      [<0.24484.16>,
       {basic_consume,false,<0.24482.16>,<0.24481.16>,true,
           <<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>,false,none,
           {'basic.consume_ok',<<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>}},
       infinity]}},
 [{gen_server2,call,3,[]},
  {rabbit_channel,'-handle_method/3-fun-1-',7,[]},
  {rabbit_misc,with_exit_handler,2,[]},
  {rabbit_channel,handle_method,3,[]},
  {rabbit_channel,handle_cast,2,[]},
  {gen_server2,handle_msg,2,[]},
  {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

=WARNING REPORT==== 15-Sep-2013::06:34:45 ===
Non-AMQP exit reason '{{{{badmatch,{error,not_found}},
                         [{rabbit_amqqueue_process,i,2,[]},
                          {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
                          {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
                          {rabbit_amqqueue_process,emit_stats,2,[]},
                          {rabbit_event,if_enabled,3,[]},
                          {rabbit_amqqueue_process,declare,3,[]},
                          {gen_server2,handle_msg,2,[]},
                          {proc_lib,init_p_do_apply,3,
                              [{file,"proc_lib.erl"},{line,239}]}]},
                        {gen_server2,call,
                            [<0.24484.16>,
                             {basic_consume,false,<0.24482.16>,<0.24481.16>,
                                 true,<<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>,
                                 false,none,
                                 {'basic.consume_ok',
                                     <<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>}},
                             infinity]}},
                       [{gen_server2,call,3,[]},
                        {rabbit_channel,'-handle_method/3-fun-1-',7,[]},
                        {rabbit_misc,with_exit_handler,2,[]},
                        {rabbit_channel,handle_method,3,[]},
                        {rabbit_channel,handle_cast,2,[]},
                        {gen_server2,handle_msg,2,[]},
                        {proc_lib,init_p_do_apply,3,
                            [{file,"proc_lib.erl"},{line,239}]}]}'

=INFO REPORT==== 15-Sep-2013::06:34:45 ===
closing AMQP connection <0.24463.16> (10.100.183.10:55948 -> 10.100.183.75:5672)

=WARNING REPORT==== 15-Sep-2013::06:34:47 ===
memory resource limit alarm set on node rabbit at SJPERMQ04.

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************








-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130917/145eeaca/attachment.htm>


More information about the rabbitmq-discuss mailing list