[rabbitmq-discuss] Loss of durable queues on HA sync

Simon MacMullen simon at rabbitmq.com
Mon Jul 28 19:05:14 BST 2014


Thanks for this. This is not a known issue, and not something we've seen 
before.

In all cases it looks like a queue process crashed because it received a 
message intended for a GM process. That should never be able to happen.

Could you send me full logs from all nodes around the time of the crash? 
It would be useful to see in what order things happened, if anything led 
up to this, when nodes restarted for autoheal (if any autoheal 
happened), etc.

Cheers, Simon

On 28/07/14 17:40, Miller Andrew wrote:
> During some server reboots at our organization over the weekend, several
> nodes in our non-Prod cluster had some crashes, and in the end, 7 of 84
> durable queues in the cluster were gone.  All nodes are running RabbitMQ
> 3.3.3 on Erlang R16B03-1.  This is a 3-node cluster, with
> cluster_partition_handling = autoheal.  I'm curious if this is a known
> issue with autoheal?  I tried recreating the issue this morning by
> recreating the series of events, but was unable to.  Our other 2-node
> non-Prod cluster, with no cluster_partition_handling, was also
> rebootted, but didn't have any problems.
>
> Thanks,
>
> Andrew Miller
>
> 08:12:01 - Node DIX-D-EVGAPP-02 (a disk node) saw node STR-D-EVGAPP-02
> (a memory node) go down (which had been the master node for many
> queues), and was promoted to master for 75 queues, including those which
> later would disappear.
>
> 08:13:30 - Node DIX-D-EVGAPP-02 saw node STR-D-EVGAPP-02 come back, and
> began synchronizing back.  Most queues synchronized fine, but 7 crashed
> with errors similar to the following - and each of these queues were
> gone after this.  The third node (another disk node) doesn't have any
> crashes in its log around this time.
>
> =ERROR REPORT==== 27-Jul-2014::08:13:30 ===
>
> ** Generic server <0.332.0> terminating
>
> ** Last message in was {add_on_right,{20,<4967.323.0>}}
>
> ** When Server state == {q,
>
>                           {amqqueue,
>
>                            {resource,<<"eventing">>,queue,
>
>                             <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                            true,false,none,
>
>                            [{<<"x-dead-letter-exchange">>,longstr,
>
>                              <<"EnterpriseEventDeadLetter">>},
>
>                             {<<"x-dead-letter-routing-key">>,longstr,
>
>                              <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
>                            <0.332.0>,
>
>                            [<8174.310.0>],
>
>                            [<8174.310.0>],
>
>                            [{vhost,<<"eventing">>},
>
>                             {name,<<"ha-all">>},
>
>                             {pattern,<<"^.*">>},
>
>                             {'apply-to',<<"queues">>},
>
>                             {definition,
>
>                              [{<<"ha-mode">>,<<"all">>},
>
>                               {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                             {priority,0}],
>
>
> [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
>
>                            []},
>
>                           none,false,rabbit_mirror_queue_master,
>
>                           {state,
>
>                            {resource,<<"eventing">>,queue,
>
>                             <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                            <0.333.0>,<0.4223.0>,rabbit_variable_queue,
>
>                            {vqstate,
>
>                             {0,{[],[]}},
>
>                             {0,{[],[]}},
>
>                             {delta,undefined,0,undefined},
>
>                             {0,{[],[]}},
>
>                             {0,{[],[]}},
>
>                             0,
>
>                             {0,nil},
>
>                             {0,nil},
>
>                             {qistate,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
>                              {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                  []}}},
>
>                               []},
>
>                              undefined,0,65536,
>
>                              #Fun<rabbit_variable_queue.2.51560583>,
>
>                              {0,nil}},
>
>                             {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,222,
>
>                                 13,179,107>>,
>
>                               {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                  []}}},
>
>                               {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                               rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>                               <0.266.0>,315468,307274,319565,323662},
>
>                              {client_msstate,msg_store_transient,
>
>
>                   <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
>                                 129,171,73>>,
>
>                               {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                  []}}},
>
>                               {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                               rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>                               <0.261.0>,294983,286789,299080,303177}},
>
>                             true,0,0,0,infinity,0,0,0,0,0,
>
>                             {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
>                             {0,nil},
>
>                             {0,nil},
>
>                             {0,nil},
>
>                             {0,nil},
>
>                             0,0},
>
>                            {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                               []}}},
>
>                            [],
>
>                            {set,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                               []}}}},
>
>
> {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
>                           undefined,undefined,undefined,undefined,
>
>                           {state,fine,5000,undefined},
>
>                           {0,nil},
>
>                           undefined,undefined,undefined,
>
>                           {state,
>
>                            {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                               []}}},
>
>                            delegate},
>
>                           {resource,<<"eventing">>,exchange,
>
>                            <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,
>
>                           running}
>
> ** Reason for termination ==
>
> ** {function_clause,
>
>         [{rabbit_amqqueue_process,handle_call,
>
>              [{add_on_right,{20,<4967.323.0>}},
>
>               {<4967.323.0>,#Ref<4967.0.0.2348>},
>
>               {q,{amqqueue,
>
>                      {resource,<<"eventing">>,queue,
>
>                          <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                      true,false,none,
>
>                      [{<<"x-dead-letter-exchange">>,longstr,
>
>                        <<"EnterpriseEventDeadLetter">>},
>
>                       {<<"x-dead-letter-routing-key">>,longstr,
>
>                        <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
>                      <0.332.0>,
>
>                      [<8174.310.0>],
>
>                      [<8174.310.0>],
>
>                      [{vhost,<<"eventing">>},
>
>                       {name,<<"ha-all">>},
>
>                       {pattern,<<"^.*">>},
>
>                       {'apply-to',<<"queues">>},
>
>                       {definition,
>
>                           [{<<"ha-mode">>,<<"all">>},
>
>                            {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                       {priority,0}],
>
>                      [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
>
>                      []},
>
>                  none,false,rabbit_mirror_queue_master,
>
>                  {state,
>
>                      {resource,<<"eventing">>,queue,
>
>                          <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                      <0.333.0>,<0.4223.0>,rabbit_variable_queue,
>
>                      {vqstate,
>
>                          {0,{[],[]}},
>
>                          {0,{[],[]}},
>
>                          {delta,undefined,0,undefined},
>
>                          {0,{[],[]}},
>
>                          {0,{[],[]}},
>
>                          0,
>
>                          {0,nil},
>
>                          {0,nil},
>
>                          {qistate,
>
>
>        "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
>                              {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]}}},
>
>                               []},
>
>                              undefined,0,65536,
>
>                              #Fun<rabbit_variable_queue.2.51560583>,
>
>                              {0,nil}},
>
>                          {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,222,
>
>                                 13,179,107>>,
>
>                               {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]}}},
>
>                               {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                               rabbit_msg_store_ets_index,
>
>
>                        "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>                               <0.266.0>,315468,307274,319565,323662},
>
>                           {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
>                                 129,171,73>>,
>
>                               {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]}}},
>
>                               {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                               rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>                               <0.261.0>,294983,286789,299080,303177}},
>
>                          true,0,0,0,infinity,0,0,0,0,0,
>
>                          {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
>                          {0,nil},
>
>                          {0,nil},
>
>                          {0,nil},
>
>                          {0,nil},
>
>                          0,0},
>
>                      {dict,0,16,16,8,80,48,
>
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                      [],
>
>                      {set,0,16,16,8,80,48,
>
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
>                  {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
>                  undefined,undefined,undefined,undefined,
>
>                  {state,fine,5000,undefined},
>
>                  {0,nil},
>
>                  undefined,undefined,undefined,
>
>                  {state,
>
>                      {dict,0,16,16,8,80,48,
>
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                      delegate},
>
>                  {resource,<<"eventing">>,exchange,
>
>                      <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
>              []},
>
>          {gen_server2,handle_msg,2,[]},
>
>          {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
>
> 08:13:30 - Node STR-D-EVGAPP-02 also recorded crashes involving the same
> queues:
>
> =ERROR REPORT==== 27-Jul-2014::08:13:30 ===
>
> ** Generic server <0.323.0> terminating
>
> ** Last message in was {'$gen_cast',join}
>
> ** When Server state == {state,
>
>                              {20,<0.323.0>},
>
>                              {{20,<0.323.0>},undefined},
>
>                              {{20,<0.323.0>},undefined},
>
>                              {resource,<<"eventing">>,queue,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
>
>
> rabbit_mirror_queue_slave,undefined,-1,undefined,
>
>                              [<0.322.0>],
>
>                              {[],[]},
>
>                              [],0,undefined,
>
>                              #Fun<rabbit_misc.execute_mnesia_transaction.1>}
>
> ** Reason for termination ==
>
> ** {{function_clause,
>
>          [{rabbit_amqqueue_process,handle_call,
>
>               [{add_on_right,{20,<0.323.0>}},
>
>                {<0.323.0>,#Ref<0.0.0.2348>},
>
>                {q,{amqqueue,
>
>                       {resource,<<"eventing">>,queue,
>
>                           <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                       true,false,none,
>
>                       [{<<"x-dead-letter-exchange">>,longstr,
>
>                         <<"EnterpriseEventDeadLetter">>},
>
>                        {<<"x-dead-letter-routing-key">>,longstr,
>
>                         <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
>                       <5297.332.0>,
>
>                       [<5251.310.0>],
>
>                       [<5251.310.0>],
>
>                       [{vhost,<<"eventing">>},
>
>                        {name,<<"ha-all">>},
>
>                        {pattern,<<"^.*">>},
>
>                        {'apply-to',<<"queues">>},
>
>                        {definition,
>
>                            [{<<"ha-mode">>,<<"all">>},
>
>                             {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                        {priority,0}],
>
>
> [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
>
>                       []},
>
>                   none,false,rabbit_mirror_queue_master,
>
>                   {state,
>
>                       {resource,<<"eventing">>,queue,
>
>                           <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                       <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
>
>                       {vqstate,
>
>                           {0,{[],[]}},
>
>                           {0,{[],[]}},
>
>                           {delta,undefined,0,undefined},
>
>                           {0,{[],[]}},
>
>                           {0,{[],[]}},
>
>                           0,
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {qistate,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
>                               {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                []},
>
>                               undefined,0,65536,
>
>                               #Fun<rabbit_variable_queue.2.51560583>,
>
>                               {0,nil}},
>
>                           {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,
>
>                                  222,13,179,107>>,
>
>                                {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                                rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>                                <5297.266.0>,315468,307274,319565,323662},
>
>                            {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
>                                  129,171,73>>,
>
>                                {dict,0,16,16,8,80,48,
>
>
>            {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                                rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>                                <5297.261.0>,294983,286789,299080,303177}},
>
>                           true,0,0,0,infinity,0,0,0,0,0,
>
>                           {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           0,0},
>
>                       {dict,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                       [],
>
>                       {set,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
>                   {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
>                   undefined,undefined,undefined,undefined,
>
>                   {state,fine,5000,undefined},
>
>                   {0,nil},
>
>                   undefined,undefined,undefined,
>
>                   {state,
>
>                       {dict,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                       delegate},
>
>                   {resource,<<"eventing">>,exchange,
>
>                       <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
>               []},
>
>           {gen_server2,handle_msg,2,[]},
>
>           {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
>
>
> {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}
>
> 08:13:31 - And again from node STR-D-EVGAPP-02:
>
> =ERROR REPORT==== 27-Jul-2014::08:13:31 ===
>
> ** Generic server <0.322.0> terminating
>
> ** Last message in was {'EXIT',<0.323.0>,
>
>                          {{function_clause,
>
>                            [{rabbit_amqqueue_process,handle_call,
>
>                              [{add_on_right,{20,<0.323.0>}},
>
>                               {<0.323.0>,#Ref<0.0.0.2348>},
>
>                               {q,
>
>                                {amqqueue,
>
>                                 {resource,<<"eventing">>,queue,
>
>                                  <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                                 true,false,none,
>
>                                 [{<<"x-dead-letter-exchange">>,longstr,
>
>                                   <<"EnterpriseEventDeadLetter">>},
>
>                                  {<<"x-dead-letter-routing-key">>,longstr,
>
>                                   <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
>                                 <5297.332.0>,
>
>                                 [<5251.310.0>],
>
>                                 [<5251.310.0>],
>
>                                 [{vhost,<<"eventing">>},
>
>                                  {name,<<"ha-all">>},
>
>                                  {pattern,<<"^.*">>},
>
>                                  {'apply-to',<<"queues">>},
>
>                                  {definition,
>
>                                   [{<<"ha-mode">>,<<"all">>},
>
>                                    {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                                  {priority,0}],
>
>                                 [{<5251.311.0>,<5251.310.0>},
>
>                                  {<5297.333.0>,<5297.332.0>}],
>
>                                 []},
>
>                                none,false,rabbit_mirror_queue_master,
>
>                                {state,
>
>                                 {resource,<<"eventing">>,queue,
>
>                                  <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                                 <5297.333.0>,<5297.4223.0>,
>
>                                 rabbit_variable_queue,
>
>                                 {vqstate,
>
>                                  {0,{[],[]}},
>
>                                  {0,{[],[]}},
>
>                                  {delta,undefined,0,undefined},
>
>                                  {0,{[],[]}},
>
>                                  {0,{[],[]}},
>
>                                  0,
>
>                                  {0,nil},
>
>                                  {0,nil},
>
>                                  {qistate,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
>                                   {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                       [],[],[]}}},
>
>                                    []},
>
>                                   undefined,0,65536,
>
>                                   #Fun<rabbit_variable_queue.2.51560583>,
>
>                                   {0,nil}},
>
>                                  {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,
>
>                                      159,222,13,179,107>>,
>
>                                    {dict,0,16,16,8,80,48,
>
>
>                      {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                       [],[],[]}}},
>
>                                    {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                                    rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>
> <5297.266.0>,315468,307274,319565,323662},
>
>                                   {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,
>
>                                      254,129,171,73>>,
>
>                                    {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                       [],[],[]}}},
>
>                                    {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                                    rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>
> <5297.261.0>,294983,286789,299080,303177}},
>
>                                  true,0,0,0,infinity,0,0,0,0,0,
>
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
>                                  {0,nil},
>
>                                  {0,nil},
>
>                                  {0,nil},
>
>                                  {0,nil},
>
>                                  0,0},
>
>                                 {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                   []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]}}},
>
>                                 [],
>
>                                 {set,0,16,16,8,80,48,
>
>
>                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                   []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]}}}},
>
>                                {state,
>
>                                 {queue,[],[],0},
>
>                                 {active,1406466722448182,1.0}},
>
>                                undefined,undefined,undefined,undefined,
>
>                                {state,fine,5000,undefined},
>
>                                {0,nil},
>
>                                undefined,undefined,undefined,
>
>                                {state,
>
>                                 {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                   []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]}}},
>
>                                 delegate},
>
>                                {resource,<<"eventing">>,exchange,
>
>                                 <<"EnterpriseEventDeadLetter">>},
>
>                                <<"SalesCenter2.BPClaims_ClosedClaim">>,
>
>                                undefined,2,running}],
>
>                              []},
>
>                             {gen_server2,handle_msg,2,[]},
>
>                             {proc_lib,wake_up,3,
>
>                              [{file,"proc_lib.erl"},{line,249}]}]},
>
>                           {gen_server2,call,
>
>                            [<5297.332.0>,
>
>                             {add_on_right,{20,<0.323.0>}},
>
>                             infinity]}}}
>
> ** When Server state == {state,
>
>                           {amqqueue,
>
>                            {resource,<<"eventing">>,queue,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
>
>                            true,false,none,
>
>                            [{<<"x-dead-letter-exchange">>,longstr,
>
>                              <<"EnterpriseEventDeadLetter">>},
>
>                             {<<"x-dead-letter-routing-key">>,longstr,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>}],
>
>                            <5251.335.0>,
>
>                            [<5297.331.0>],
>
>                            [<5297.331.0>],
>
>                            [{vhost,<<"eventing">>},
>
>                             {name,<<"ha-all">>},
>
>                             {pattern,<<"^.*">>},
>
>                             {'apply-to',<<"queues">>},
>
>                             {definition,
>
>                              [{<<"ha-mode">>,<<"all">>},
>
>                               {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                             {priority,0}],
>
>                            [{<5297.332.0>,<5297.331.0>},
>
>                             {<5251.336.0>,<5251.335.0>}],
>
>                            []},
>
>                           <0.323.0>,rabbit_variable_queue,
>
>                           {vqstate,
>
>                            {0,{[],[]}},
>
>                            {0,{[],[]}},
>
>                            {delta,undefined,0,undefined},
>
>                            {0,{[],[]}},
>
>                            {0,{[],[]}},
>
>                            0,
>
>                            {0,nil},
>
>                            {0,nil},
>
>                            {qistate,
>
>
> "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/queues/11YNGR2UMDIGSF4OWRUUJ2HUX",
>
>                             {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []}}},
>
>                              []},
>
>                             undefined,0,65536,
>
>                             #Fun<rabbit_variable_queue.2.51560583>,
>
>                             {0,nil}},
>
>                            {{client_msstate,msg_store_persistent,
>
>
> <<142,49,10,175,39,218,164,74,2,215,253,147,201,
>
>                                168,141,2>>,
>
>                              {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []}}},
>
>                              {state,303177,
>
>
> "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                              rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>                              <0.251.0>,307274,299080,311371,315468},
>
>                             {client_msstate,msg_store_transient,
>
>
> <<18,152,170,252,55,149,243,37,252,234,131,180,53,
>
>                                66,14,182>>,
>
>                              {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []}}},
>
>                              {state,282692,
>
>
> "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                              rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_transient",
>
>                              <0.246.0>,286789,278595,290886,294983}},
>
>                            true,0,0,0,infinity,0,0,0,0,0,
>
>                            {rates,0.0,0.0,0.0,0.0,{1406,466811,43211}},
>
>                            {0,nil},
>
>                            {0,nil},
>
>                            {0,nil},
>
>                            {0,nil},
>
>                            0,0},
>
>                           undefined,undefined,
>
>                           {dict,0,16,16,8,80,48,
>
>
>    {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                           {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                           {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                           {state,
>
>                            {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                               []}}},
>
>                            delegate},
>
>                           undefined}
>
> ** Reason for termination ==
>
> ** {{function_clause,
>
>          [{rabbit_amqqueue_process,handle_call,
>
>               [{add_on_right,{20,<0.323.0>}},
>
>                {<0.323.0>,#Ref<0.0.0.2348>},
>
>                {q,{amqqueue,
>
>                       {resource,<<"eventing">>,queue,
>
>                           <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                       true,false,none,
>
>                       [{<<"x-dead-letter-exchange">>,longstr,
>
>                         <<"EnterpriseEventDeadLetter">>},
>
>                        {<<"x-dead-letter-routing-key">>,longstr,
>
>                         <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
>                       <5297.332.0>,
>
>                       [<5251.310.0>],
>
>                       [<5251.310.0>],
>
>                       [{vhost,<<"eventing">>},
>
>                        {name,<<"ha-all">>},
>
>                        {pattern,<<"^.*">>},
>
>                        {'apply-to',<<"queues">>},
>
>                        {definition,
>
>                            [{<<"ha-mode">>,<<"all">>},
>
>                             {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                        {priority,0}],
>
>
> [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
>
>                       []},
>
>                   none,false,rabbit_mirror_queue_master,
>
>                   {state,
>
>                       {resource,<<"eventing">>,queue,
>
>                           <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                       <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
>
>                       {vqstate,
>
>                           {0,{[],[]}},
>
>                           {0,{[],[]}},
>
>                           {delta,undefined,0,undefined},
>
>                           {0,{[],[]}},
>
>                           {0,{[],[]}},
>
>                           0,
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {qistate,
>
>
>           "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
>                               {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                []},
>
>                               undefined,0,65536,
>
>                               #Fun<rabbit_variable_queue.2.51560583>,
>
>                               {0,nil}},
>
>                           {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,
>
>                                  222,13,179,107>>,
>
>                                {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                                rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>                                <5297.266.0>,315468,307274,319565,323662},
>
>                            {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
>                                  129,171,73>>,
>
>                                {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                {state,290886,
>
>
>        "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                                rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>                                <5297.261.0>,294983,286789,299080,303177}},
>
>                           true,0,0,0,infinity,0,0,0,0,0,
>
>                           {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           0,0},
>
>                       {dict,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                       [],
>
>                       {set,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
>                   {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
>                   undefined,undefined,undefined,undefined,
>
>                   {state,fine,5000,undefined},
>
>                   {0,nil},
>
>                   undefined,undefined,undefined,
>
>                   {state,
>
>                       {dict,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                       delegate},
>
>                   {resource,<<"eventing">>,exchange,
>
>                       <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
>               []},
>
>           {gen_server2,handle_msg,2,[]},
>
>           {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
>
>
> {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}
>
> This e-mail is confidential.  If you are not the intended recipient, you
> must not disclose or use the information contained in it.  If you have
> received this e-mail in error, please tell us immediately by return
> e-mail and delete the document. No recipient may use the information in
> this e-mail in violation of any civil or criminal statute. Sentry
> disclaims all liability for any unauthorized uses of this e-mail or its
> contents. Sentry accepts no liability or responsibility for any damage
> caused by any virus transmitted with this e-mail.
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
> please subscribe to the new list!
>
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>




More information about the rabbitmq-discuss mailing list