[rabbitmq-discuss] Loss of durable queues on HA sync

Miller Andrew Andrew.Miller at sentry.com
Mon Jul 28 17:40:30 BST 2014


During some server reboots at our organization over the weekend, several nodes in our non-Prod cluster had some crashes, and in the end, 7 of 84 durable queues in the cluster were gone.  All nodes are running RabbitMQ 3.3.3 on Erlang R16B03-1.  This is a 3-node cluster, with cluster_partition_handling = autoheal.  I'm curious if this is a known issue with autoheal?  I tried recreating the issue this morning by recreating the series of events, but was unable to.  Our other 2-node non-Prod cluster, with no cluster_partition_handling, was also rebootted, but didn't have any problems.

Thanks,
Andrew Miller

08:12:01 - Node DIX-D-EVGAPP-02 (a disk node) saw node STR-D-EVGAPP-02 (a memory node) go down (which had been the master node for many queues), and was promoted to master for 75 queues, including those which later would disappear.

08:13:30 - Node DIX-D-EVGAPP-02 saw node STR-D-EVGAPP-02 come back, and began synchronizing back.  Most queues synchronized fine, but 7 crashed with errors similar to the following - and each of these queues were gone after this.  The third node (another disk node) doesn't have any crashes in its log around this time.

=ERROR REPORT==== 27-Jul-2014::08:13:30 ===
** Generic server <0.332.0> terminating
** Last message in was {add_on_right,{20,<4967.323.0>}}
** When Server state == {q,
                         {amqqueue,
                          {resource,<<"eventing">>,queue,
                           <<"SalesCenter2.BPClaims_ClosedClaim">>},
                          true,false,none,
                          [{<<"x-dead-letter-exchange">>,longstr,
                            <<"EnterpriseEventDeadLetter">>},
                           {<<"x-dead-letter-routing-key">>,longstr,
                            <<"SalesCenter2.BPClaims_ClosedClaim">>}],
                          <0.332.0>,
                          [<8174.310.0>],
                          [<8174.310.0>],
                          [{vhost,<<"eventing">>},
                           {name,<<"ha-all">>},
                           {pattern,<<"^.*">>},
                           {'apply-to',<<"queues">>},
                           {definition,
                            [{<<"ha-mode">>,<<"all">>},
                             {<<"ha-sync-mode">>,<<"automatic">>}]},
                           {priority,0}],
                          [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
                          []},
                         none,false,rabbit_mirror_queue_master,
                         {state,
                          {resource,<<"eventing">>,queue,
                           <<"SalesCenter2.BPClaims_ClosedClaim">>},
                          <0.333.0>,<0.4223.0>,rabbit_variable_queue,
                          {vqstate,
                           {0,{[],[]}},
                           {0,{[],[]}},
                           {delta,undefined,0,undefined},
                           {0,{[],[]}},
                           {0,{[],[]}},
                           0,
                           {0,nil},
                           {0,nil},
                           {qistate,
                            "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
                            {{dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},
                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                             []},
                            undefined,0,65536,
                            #Fun<rabbit_variable_queue.2.51560583>,
                            {0,nil}},
                           {{client_msstate,msg_store_persistent,
                             <<26,131,31,121,222,176,41,144,168,192,42,159,222,
                               13,179,107>>,
                             {dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},
                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                             {state,311371,
                              "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
                             rabbit_msg_store_ets_index,
                             "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
                             <0.266.0>,315468,307274,319565,323662},
                            {client_msstate,msg_store_transient,
                             <<32,49,130,217,86,65,129,214,32,140,24,34,254,
                               129,171,73>>,
                             {dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},
                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                             {state,290886,
                              "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
                             rabbit_msg_store_ets_index,
                             "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
                             <0.261.0>,294983,286789,299080,303177}},
                           true,0,0,0,infinity,0,0,0,0,0,
                           {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
                           {0,nil},
                           {0,nil},
                           {0,nil},
                           {0,nil},
                           0,0},
                          {dict,0,16,16,8,80,48,
                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                             []}}},
                          [],
                          {set,0,16,16,8,80,48,
                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                             []}}}},
                         {state,{queue,[],[],0},{active,1406466722448182,1.0}},
                         undefined,undefined,undefined,undefined,
                         {state,fine,5000,undefined},
                         {0,nil},
                         undefined,undefined,undefined,
                         {state,
                          {dict,0,16,16,8,80,48,
                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                             []}}},
                          delegate},
                         {resource,<<"eventing">>,exchange,
                          <<"EnterpriseEventDeadLetter">>},
                         <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,
                         running}
** Reason for termination ==
** {function_clause,
       [{rabbit_amqqueue_process,handle_call,
            [{add_on_right,{20,<4967.323.0>}},
             {<4967.323.0>,#Ref<4967.0.0.2348>},
             {q,{amqqueue,
                    {resource,<<"eventing">>,queue,
                        <<"SalesCenter2.BPClaims_ClosedClaim">>},
                    true,false,none,
                    [{<<"x-dead-letter-exchange">>,longstr,
                      <<"EnterpriseEventDeadLetter">>},
                     {<<"x-dead-letter-routing-key">>,longstr,
                      <<"SalesCenter2.BPClaims_ClosedClaim">>}],
                    <0.332.0>,
                    [<8174.310.0>],
                    [<8174.310.0>],
                    [{vhost,<<"eventing">>},
                     {name,<<"ha-all">>},
                     {pattern,<<"^.*">>},
                     {'apply-to',<<"queues">>},
                     {definition,
                         [{<<"ha-mode">>,<<"all">>},
                          {<<"ha-sync-mode">>,<<"automatic">>}]},
                     {priority,0}],
                    [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
                    []},
                none,false,rabbit_mirror_queue_master,
                {state,
                    {resource,<<"eventing">>,queue,
                        <<"SalesCenter2.BPClaims_ClosedClaim">>},
                    <0.333.0>,<0.4223.0>,rabbit_variable_queue,
                    {vqstate,
                        {0,{[],[]}},
                        {0,{[],[]}},
                        {delta,undefined,0,undefined},
                        {0,{[],[]}},
                        {0,{[],[]}},
                        0,
                        {0,nil},
                        {0,nil},
                        {qistate,
                            "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
                            {{dict,0,16,16,8,80,48,
                                 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]},
                                 {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]}}},
                             []},
                            undefined,0,65536,
                            #Fun<rabbit_variable_queue.2.51560583>,
                            {0,nil}},
                        {{client_msstate,msg_store_persistent,
                             <<26,131,31,121,222,176,41,144,168,192,42,159,222,
                               13,179,107>>,
                             {dict,0,16,16,8,80,48,
                                 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]},
                                 {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]}}},
                             {state,311371,
                                 "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
                             rabbit_msg_store_ets_index,
                             "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
                             <0.266.0>,315468,307274,319565,323662},
                         {client_msstate,msg_store_transient,
                             <<32,49,130,217,86,65,129,214,32,140,24,34,254,
                               129,171,73>>,
                             {dict,0,16,16,8,80,48,
                                 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]},
                                 {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]}}},
                             {state,290886,
                                 "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
                             rabbit_msg_store_ets_index,
                             "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
                             <0.261.0>,294983,286789,299080,303177}},
                        true,0,0,0,infinity,0,0,0,0,0,
                        {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
                        {0,nil},
                        {0,nil},
                        {0,nil},
                        {0,nil},
                        0,0},
                    {dict,0,16,16,8,80,48,
                        {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                        {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                    [],
                    {set,0,16,16,8,80,48,
                        {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                        {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
                {state,{queue,[],[],0},{active,1406466722448182,1.0}},
                undefined,undefined,undefined,undefined,
                {state,fine,5000,undefined},
                {0,nil},
                undefined,undefined,undefined,
                {state,
                    {dict,0,16,16,8,80,48,
                        {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                        {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                    delegate},
                {resource,<<"eventing">>,exchange,
                    <<"EnterpriseEventDeadLetter">>},
                <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
            []},
        {gen_server2,handle_msg,2,[]},
        {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}


08:13:30 - Node STR-D-EVGAPP-02 also recorded crashes involving the same queues:

=ERROR REPORT==== 27-Jul-2014::08:13:30 ===
** Generic server <0.323.0> terminating
** Last message in was {'$gen_cast',join}
** When Server state == {state,
                            {20,<0.323.0>},
                            {{20,<0.323.0>},undefined},
                            {{20,<0.323.0>},undefined},
                            {resource,<<"eventing">>,queue,
                                <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
                            rabbit_mirror_queue_slave,undefined,-1,undefined,
                            [<0.322.0>],
                            {[],[]},
                            [],0,undefined,
                            #Fun<rabbit_misc.execute_mnesia_transaction.1>}
** Reason for termination ==
** {{function_clause,
        [{rabbit_amqqueue_process,handle_call,
             [{add_on_right,{20,<0.323.0>}},
              {<0.323.0>,#Ref<0.0.0.2348>},
              {q,{amqqueue,
                     {resource,<<"eventing">>,queue,
                         <<"SalesCenter2.BPClaims_ClosedClaim">>},
                     true,false,none,
                     [{<<"x-dead-letter-exchange">>,longstr,
                       <<"EnterpriseEventDeadLetter">>},
                      {<<"x-dead-letter-routing-key">>,longstr,
                       <<"SalesCenter2.BPClaims_ClosedClaim">>}],
                     <5297.332.0>,
                     [<5251.310.0>],
                     [<5251.310.0>],
                     [{vhost,<<"eventing">>},
                      {name,<<"ha-all">>},
                      {pattern,<<"^.*">>},
                      {'apply-to',<<"queues">>},
                      {definition,
                          [{<<"ha-mode">>,<<"all">>},
                           {<<"ha-sync-mode">>,<<"automatic">>}]},
                      {priority,0}],
                     [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
                     []},
                 none,false,rabbit_mirror_queue_master,
                 {state,
                     {resource,<<"eventing">>,queue,
                         <<"SalesCenter2.BPClaims_ClosedClaim">>},
                     <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
                     {vqstate,
                         {0,{[],[]}},
                         {0,{[],[]}},
                         {delta,undefined,0,undefined},
                         {0,{[],[]}},
                         {0,{[],[]}},
                         0,
                         {0,nil},
                         {0,nil},
                         {qistate,
                             "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
                             {{dict,0,16,16,8,80,48,
                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]},
                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]}}},
                              []},
                             undefined,0,65536,
                             #Fun<rabbit_variable_queue.2.51560583>,
                             {0,nil}},
                         {{client_msstate,msg_store_persistent,
                              <<26,131,31,121,222,176,41,144,168,192,42,159,
                                222,13,179,107>>,
                              {dict,0,16,16,8,80,48,
                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]},
                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]}}},
                              {state,311371,
                                  "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
                              rabbit_msg_store_ets_index,
                              "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
                              <5297.266.0>,315468,307274,319565,323662},
                          {client_msstate,msg_store_transient,
                              <<32,49,130,217,86,65,129,214,32,140,24,34,254,
                                129,171,73>>,
                              {dict,0,16,16,8,80,48,
                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]},
                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]}}},
                              {state,290886,
                                  "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
                              rabbit_msg_store_ets_index,
                              "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
                              <5297.261.0>,294983,286789,299080,303177}},
                         true,0,0,0,infinity,0,0,0,0,0,
                         {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
                         {0,nil},
                         {0,nil},
                         {0,nil},
                         {0,nil},
                         0,0},
                     {dict,0,16,16,8,80,48,
                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                     [],
                     {set,0,16,16,8,80,48,
                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
                 {state,{queue,[],[],0},{active,1406466722448182,1.0}},
                 undefined,undefined,undefined,undefined,
                 {state,fine,5000,undefined},
                 {0,nil},
                 undefined,undefined,undefined,
                 {state,
                     {dict,0,16,16,8,80,48,
                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                     delegate},
                 {resource,<<"eventing">>,exchange,
                     <<"EnterpriseEventDeadLetter">>},
                 <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
             []},
         {gen_server2,handle_msg,2,[]},
         {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
    {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}

08:13:31 - And again from node STR-D-EVGAPP-02:

=ERROR REPORT==== 27-Jul-2014::08:13:31 ===
** Generic server <0.322.0> terminating
** Last message in was {'EXIT',<0.323.0>,
                        {{function_clause,
                          [{rabbit_amqqueue_process,handle_call,
                            [{add_on_right,{20,<0.323.0>}},
                             {<0.323.0>,#Ref<0.0.0.2348>},
                             {q,
                              {amqqueue,
                               {resource,<<"eventing">>,queue,
                                <<"SalesCenter2.BPClaims_ClosedClaim">>},
                               true,false,none,
                               [{<<"x-dead-letter-exchange">>,longstr,
                                 <<"EnterpriseEventDeadLetter">>},
                                {<<"x-dead-letter-routing-key">>,longstr,
                                 <<"SalesCenter2.BPClaims_ClosedClaim">>}],
                               <5297.332.0>,
                               [<5251.310.0>],
                               [<5251.310.0>],
                               [{vhost,<<"eventing">>},
                                {name,<<"ha-all">>},
                                {pattern,<<"^.*">>},
                                {'apply-to',<<"queues">>},
                                {definition,
                                 [{<<"ha-mode">>,<<"all">>},
                                  {<<"ha-sync-mode">>,<<"automatic">>}]},
                                {priority,0}],
                               [{<5251.311.0>,<5251.310.0>},
                                {<5297.333.0>,<5297.332.0>}],
                               []},
                              none,false,rabbit_mirror_queue_master,
                              {state,
                               {resource,<<"eventing">>,queue,
                                <<"SalesCenter2.BPClaims_ClosedClaim">>},
                               <5297.333.0>,<5297.4223.0>,
                               rabbit_variable_queue,
                               {vqstate,
                                {0,{[],[]}},
                                {0,{[],[]}},
                                {delta,undefined,0,undefined},
                                {0,{[],[]}},
                                {0,{[],[]}},
                                0,
                                {0,nil},
                                {0,nil},
                                {qistate,
                                 "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
                                 {{dict,0,16,16,8,80,48,
                                   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]},
                                   {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                     [],[],[]}}},
                                  []},
                                 undefined,0,65536,
                                 #Fun<rabbit_variable_queue.2.51560583>,
                                 {0,nil}},
                                {{client_msstate,msg_store_persistent,
                                  <<26,131,31,121,222,176,41,144,168,192,42,
                                    159,222,13,179,107>>,
                                  {dict,0,16,16,8,80,48,
                                   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]},
                                   {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                     [],[],[]}}},
                                  {state,311371,
                                   "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
                                  rabbit_msg_store_ets_index,
                                  "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
                                  <5297.266.0>,315468,307274,319565,323662},
                                 {client_msstate,msg_store_transient,
                                  <<32,49,130,217,86,65,129,214,32,140,24,34,
                                    254,129,171,73>>,
                                  {dict,0,16,16,8,80,48,
                                   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]},
                                   {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                     [],[],[]}}},
                                  {state,290886,
                                   "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
                                  rabbit_msg_store_ets_index,
                                  "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
                                  <5297.261.0>,294983,286789,299080,303177}},
                                true,0,0,0,infinity,0,0,0,0,0,
                                {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
                                {0,nil},
                                {0,nil},
                                {0,nil},
                                {0,nil},
                                0,0},
                               {dict,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                               [],
                               {set,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}}},
                              {state,
                               {queue,[],[],0},
                               {active,1406466722448182,1.0}},
                              undefined,undefined,undefined,undefined,
                              {state,fine,5000,undefined},
                              {0,nil},
                              undefined,undefined,undefined,
                              {state,
                               {dict,0,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]}}},
                               delegate},
                              {resource,<<"eventing">>,exchange,
                               <<"EnterpriseEventDeadLetter">>},
                              <<"SalesCenter2.BPClaims_ClosedClaim">>,
                              undefined,2,running}],
                            []},
                           {gen_server2,handle_msg,2,[]},
                           {proc_lib,wake_up,3,
                            [{file,"proc_lib.erl"},{line,249}]}]},
                         {gen_server2,call,
                          [<5297.332.0>,
                           {add_on_right,{20,<0.323.0>}},
                           infinity]}}}
** When Server state == {state,
                         {amqqueue,
                          {resource,<<"eventing">>,queue,
                           <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
                          true,false,none,
                          [{<<"x-dead-letter-exchange">>,longstr,
                            <<"EnterpriseEventDeadLetter">>},
                           {<<"x-dead-letter-routing-key">>,longstr,
                            <<"Insight.BPClaims_ModifiedWorkNotAvailable">>}],
                          <5251.335.0>,
                          [<5297.331.0>],
                          [<5297.331.0>],
                          [{vhost,<<"eventing">>},
                           {name,<<"ha-all">>},
                           {pattern,<<"^.*">>},
                           {'apply-to',<<"queues">>},
                           {definition,
                            [{<<"ha-mode">>,<<"all">>},
                             {<<"ha-sync-mode">>,<<"automatic">>}]},
                           {priority,0}],
                          [{<5297.332.0>,<5297.331.0>},
                           {<5251.336.0>,<5251.335.0>}],
                          []},
                         <0.323.0>,rabbit_variable_queue,
                         {vqstate,
                          {0,{[],[]}},
                          {0,{[],[]}},
                          {delta,undefined,0,undefined},
                          {0,{[],[]}},
                          {0,{[],[]}},
                          0,
                          {0,nil},
                          {0,nil},
                          {qistate,
                           "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/queues/11YNGR2UMDIGSF4OWRUUJ2HUX",
                           {{dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            []},
                           undefined,0,65536,
                           #Fun<rabbit_variable_queue.2.51560583>,
                           {0,nil}},
                          {{client_msstate,msg_store_persistent,
                            <<142,49,10,175,39,218,164,74,2,215,253,147,201,
                              168,141,2>>,
                            {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,303177,
                             "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_persistent"},
                            rabbit_msg_store_ets_index,
                            "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_persistent",
                            <0.251.0>,307274,299080,311371,315468},
                           {client_msstate,msg_store_transient,
                            <<18,152,170,252,55,149,243,37,252,234,131,180,53,
                              66,14,182>>,
                            {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}},
                            {state,282692,
                             "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_transient"},
                            rabbit_msg_store_ets_index,
                            "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_transient",
                            <0.246.0>,286789,278595,290886,294983}},
                          true,0,0,0,infinity,0,0,0,0,0,
                          {rates,0.0,0.0,0.0,0.0,{1406,466811,43211}},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          {0,nil},
                          0,0},
                         undefined,undefined,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {state,
                          {dict,0,16,16,8,80,48,
                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                             []}}},
                          delegate},
                         undefined}
** Reason for termination ==
** {{function_clause,
        [{rabbit_amqqueue_process,handle_call,
             [{add_on_right,{20,<0.323.0>}},
              {<0.323.0>,#Ref<0.0.0.2348>},
              {q,{amqqueue,
                     {resource,<<"eventing">>,queue,
                         <<"SalesCenter2.BPClaims_ClosedClaim">>},
                     true,false,none,
                     [{<<"x-dead-letter-exchange">>,longstr,
                       <<"EnterpriseEventDeadLetter">>},
                      {<<"x-dead-letter-routing-key">>,longstr,
                       <<"SalesCenter2.BPClaims_ClosedClaim">>}],
                     <5297.332.0>,
                     [<5251.310.0>],
                     [<5251.310.0>],
                     [{vhost,<<"eventing">>},
                      {name,<<"ha-all">>},
                      {pattern,<<"^.*">>},
                      {'apply-to',<<"queues">>},
                      {definition,
                          [{<<"ha-mode">>,<<"all">>},
                           {<<"ha-sync-mode">>,<<"automatic">>}]},
                      {priority,0}],
                     [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
                     []},
                 none,false,rabbit_mirror_queue_master,
                 {state,
                     {resource,<<"eventing">>,queue,
                         <<"SalesCenter2.BPClaims_ClosedClaim">>},
                     <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
                     {vqstate,
                         {0,{[],[]}},
                         {0,{[],[]}},
                         {delta,undefined,0,undefined},
                         {0,{[],[]}},
                         {0,{[],[]}},
                         0,
                         {0,nil},
                         {0,nil},
                         {qistate,
                             "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
                             {{dict,0,16,16,8,80,48,
                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]},
                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]}}},
                              []},
                             undefined,0,65536,
                             #Fun<rabbit_variable_queue.2.51560583>,
                             {0,nil}},
                         {{client_msstate,msg_store_persistent,
                              <<26,131,31,121,222,176,41,144,168,192,42,159,
                                222,13,179,107>>,
                              {dict,0,16,16,8,80,48,
                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]},
                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]}}},
                              {state,311371,
                                  "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
                              rabbit_msg_store_ets_index,
                              "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
                              <5297.266.0>,315468,307274,319565,323662},
                          {client_msstate,msg_store_transient,
                              <<32,49,130,217,86,65,129,214,32,140,24,34,254,
                                129,171,73>>,
                              {dict,0,16,16,8,80,48,
                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]},
                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [],[]}}},
                              {state,290886,
                                  "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
                              rabbit_msg_store_ets_index,
                              "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
                              <5297.261.0>,294983,286789,299080,303177}},
                         true,0,0,0,infinity,0,0,0,0,0,
                         {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
                         {0,nil},
                         {0,nil},
                         {0,nil},
                         {0,nil},
                         0,0},
                     {dict,0,16,16,8,80,48,
                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                     [],
                     {set,0,16,16,8,80,48,
                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
                 {state,{queue,[],[],0},{active,1406466722448182,1.0}},
                 undefined,undefined,undefined,undefined,
                 {state,fine,5000,undefined},
                 {0,nil},
                 undefined,undefined,undefined,
                 {state,
                     {dict,0,16,16,8,80,48,
                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                     delegate},
                 {resource,<<"eventing">>,exchange,
                     <<"EnterpriseEventDeadLetter">>},
                 <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
             []},
         {gen_server2,handle_msg,2,[]},
         {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
    {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}

This e-mail is confidential.  If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this e-mail in error, please tell us immediately by return e-mail and delete the document. No recipient may use the information in this e-mail in violation of any civil or criminal statute. Sentry disclaims all liability for any unauthorized uses of this e-mail or its contents. Sentry accepts no liability or responsibility for any damage caused by any virus transmitted with this e-mail.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140728/c09ad936/attachment.html>


More information about the rabbitmq-discuss mailing list