[rabbitmq-discuss] Loss of durable queues on HA sync
Miller Andrew
Andrew.Miller at sentry.com
Mon Jul 28 17:40:30 BST 2014
During some server reboots at our organization over the weekend, several nodes in our non-Prod cluster had some crashes, and in the end, 7 of 84 durable queues in the cluster were gone. All nodes are running RabbitMQ 3.3.3 on Erlang R16B03-1. This is a 3-node cluster, with cluster_partition_handling = autoheal. I'm curious if this is a known issue with autoheal? I tried recreating the issue this morning by recreating the series of events, but was unable to. Our other 2-node non-Prod cluster, with no cluster_partition_handling, was also rebootted, but didn't have any problems.
Thanks,
Andrew Miller
08:12:01 - Node DIX-D-EVGAPP-02 (a disk node) saw node STR-D-EVGAPP-02 (a memory node) go down (which had been the master node for many queues), and was promoted to master for 75 queues, including those which later would disappear.
08:13:30 - Node DIX-D-EVGAPP-02 saw node STR-D-EVGAPP-02 come back, and began synchronizing back. Most queues synchronized fine, but 7 crashed with errors similar to the following - and each of these queues were gone after this. The third node (another disk node) doesn't have any crashes in its log around this time.
=ERROR REPORT==== 27-Jul-2014::08:13:30 ===
** Generic server <0.332.0> terminating
** Last message in was {add_on_right,{20,<4967.323.0>}}
** When Server state == {q,
{amqqueue,
{resource,<<"eventing">>,queue,
<<"SalesCenter2.BPClaims_ClosedClaim">>},
true,false,none,
[{<<"x-dead-letter-exchange">>,longstr,
<<"EnterpriseEventDeadLetter">>},
{<<"x-dead-letter-routing-key">>,longstr,
<<"SalesCenter2.BPClaims_ClosedClaim">>}],
<0.332.0>,
[<8174.310.0>],
[<8174.310.0>],
[{vhost,<<"eventing">>},
{name,<<"ha-all">>},
{pattern,<<"^.*">>},
{'apply-to',<<"queues">>},
{definition,
[{<<"ha-mode">>,<<"all">>},
{<<"ha-sync-mode">>,<<"automatic">>}]},
{priority,0}],
[{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
[]},
none,false,rabbit_mirror_queue_master,
{state,
{resource,<<"eventing">>,queue,
<<"SalesCenter2.BPClaims_ClosedClaim">>},
<0.333.0>,<0.4223.0>,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
{0,nil},
{qistate,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
[]},
undefined,0,65536,
#Fun<rabbit_variable_queue.2.51560583>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<26,131,31,121,222,176,41,144,168,192,42,159,222,
13,179,107>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,311371,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
<0.266.0>,315468,307274,319565,323662},
{client_msstate,msg_store_transient,
<<32,49,130,217,86,65,129,214,32,140,24,34,254,
129,171,73>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,290886,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
<0.261.0>,294983,286789,299080,303177}},
true,0,0,0,infinity,0,0,0,0,0,
{rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
[],
{set,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}}},
{state,{queue,[],[],0},{active,1406466722448182,1.0}},
undefined,undefined,undefined,undefined,
{state,fine,5000,undefined},
{0,nil},
undefined,undefined,undefined,
{state,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
delegate},
{resource,<<"eventing">>,exchange,
<<"EnterpriseEventDeadLetter">>},
<<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,
running}
** Reason for termination ==
** {function_clause,
[{rabbit_amqqueue_process,handle_call,
[{add_on_right,{20,<4967.323.0>}},
{<4967.323.0>,#Ref<4967.0.0.2348>},
{q,{amqqueue,
{resource,<<"eventing">>,queue,
<<"SalesCenter2.BPClaims_ClosedClaim">>},
true,false,none,
[{<<"x-dead-letter-exchange">>,longstr,
<<"EnterpriseEventDeadLetter">>},
{<<"x-dead-letter-routing-key">>,longstr,
<<"SalesCenter2.BPClaims_ClosedClaim">>}],
<0.332.0>,
[<8174.310.0>],
[<8174.310.0>],
[{vhost,<<"eventing">>},
{name,<<"ha-all">>},
{pattern,<<"^.*">>},
{'apply-to',<<"queues">>},
{definition,
[{<<"ha-mode">>,<<"all">>},
{<<"ha-sync-mode">>,<<"automatic">>}]},
{priority,0}],
[{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
[]},
none,false,rabbit_mirror_queue_master,
{state,
{resource,<<"eventing">>,queue,
<<"SalesCenter2.BPClaims_ClosedClaim">>},
<0.333.0>,<0.4223.0>,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
{0,nil},
{qistate,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
[]},
undefined,0,65536,
#Fun<rabbit_variable_queue.2.51560583>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<26,131,31,121,222,176,41,144,168,192,42,159,222,
13,179,107>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{state,311371,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
<0.266.0>,315468,307274,319565,323662},
{client_msstate,msg_store_transient,
<<32,49,130,217,86,65,129,214,32,140,24,34,254,
129,171,73>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{state,290886,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
<0.261.0>,294983,286789,299080,303177}},
true,0,0,0,infinity,0,0,0,0,0,
{rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
[],
{set,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
{state,{queue,[],[],0},{active,1406466722448182,1.0}},
undefined,undefined,undefined,undefined,
{state,fine,5000,undefined},
{0,nil},
undefined,undefined,undefined,
{state,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
delegate},
{resource,<<"eventing">>,exchange,
<<"EnterpriseEventDeadLetter">>},
<<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
08:13:30 - Node STR-D-EVGAPP-02 also recorded crashes involving the same queues:
=ERROR REPORT==== 27-Jul-2014::08:13:30 ===
** Generic server <0.323.0> terminating
** Last message in was {'$gen_cast',join}
** When Server state == {state,
{20,<0.323.0>},
{{20,<0.323.0>},undefined},
{{20,<0.323.0>},undefined},
{resource,<<"eventing">>,queue,
<<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
rabbit_mirror_queue_slave,undefined,-1,undefined,
[<0.322.0>],
{[],[]},
[],0,undefined,
#Fun<rabbit_misc.execute_mnesia_transaction.1>}
** Reason for termination ==
** {{function_clause,
[{rabbit_amqqueue_process,handle_call,
[{add_on_right,{20,<0.323.0>}},
{<0.323.0>,#Ref<0.0.0.2348>},
{q,{amqqueue,
{resource,<<"eventing">>,queue,
<<"SalesCenter2.BPClaims_ClosedClaim">>},
true,false,none,
[{<<"x-dead-letter-exchange">>,longstr,
<<"EnterpriseEventDeadLetter">>},
{<<"x-dead-letter-routing-key">>,longstr,
<<"SalesCenter2.BPClaims_ClosedClaim">>}],
<5297.332.0>,
[<5251.310.0>],
[<5251.310.0>],
[{vhost,<<"eventing">>},
{name,<<"ha-all">>},
{pattern,<<"^.*">>},
{'apply-to',<<"queues">>},
{definition,
[{<<"ha-mode">>,<<"all">>},
{<<"ha-sync-mode">>,<<"automatic">>}]},
{priority,0}],
[{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
[]},
none,false,rabbit_mirror_queue_master,
{state,
{resource,<<"eventing">>,queue,
<<"SalesCenter2.BPClaims_ClosedClaim">>},
<5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
{0,nil},
{qistate,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
[]},
undefined,0,65536,
#Fun<rabbit_variable_queue.2.51560583>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<26,131,31,121,222,176,41,144,168,192,42,159,
222,13,179,107>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{state,311371,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
<5297.266.0>,315468,307274,319565,323662},
{client_msstate,msg_store_transient,
<<32,49,130,217,86,65,129,214,32,140,24,34,254,
129,171,73>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{state,290886,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
<5297.261.0>,294983,286789,299080,303177}},
true,0,0,0,infinity,0,0,0,0,0,
{rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
[],
{set,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
{state,{queue,[],[],0},{active,1406466722448182,1.0}},
undefined,undefined,undefined,undefined,
{state,fine,5000,undefined},
{0,nil},
undefined,undefined,undefined,
{state,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
delegate},
{resource,<<"eventing">>,exchange,
<<"EnterpriseEventDeadLetter">>},
<<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
{gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}
08:13:31 - And again from node STR-D-EVGAPP-02:
=ERROR REPORT==== 27-Jul-2014::08:13:31 ===
** Generic server <0.322.0> terminating
** Last message in was {'EXIT',<0.323.0>,
{{function_clause,
[{rabbit_amqqueue_process,handle_call,
[{add_on_right,{20,<0.323.0>}},
{<0.323.0>,#Ref<0.0.0.2348>},
{q,
{amqqueue,
{resource,<<"eventing">>,queue,
<<"SalesCenter2.BPClaims_ClosedClaim">>},
true,false,none,
[{<<"x-dead-letter-exchange">>,longstr,
<<"EnterpriseEventDeadLetter">>},
{<<"x-dead-letter-routing-key">>,longstr,
<<"SalesCenter2.BPClaims_ClosedClaim">>}],
<5297.332.0>,
[<5251.310.0>],
[<5251.310.0>],
[{vhost,<<"eventing">>},
{name,<<"ha-all">>},
{pattern,<<"^.*">>},
{'apply-to',<<"queues">>},
{definition,
[{<<"ha-mode">>,<<"all">>},
{<<"ha-sync-mode">>,<<"automatic">>}]},
{priority,0}],
[{<5251.311.0>,<5251.310.0>},
{<5297.333.0>,<5297.332.0>}],
[]},
none,false,rabbit_mirror_queue_master,
{state,
{resource,<<"eventing">>,queue,
<<"SalesCenter2.BPClaims_ClosedClaim">>},
<5297.333.0>,<5297.4223.0>,
rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
{0,nil},
{qistate,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[]}}},
[]},
undefined,0,65536,
#Fun<rabbit_variable_queue.2.51560583>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<26,131,31,121,222,176,41,144,168,192,42,
159,222,13,179,107>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[]}}},
{state,311371,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
<5297.266.0>,315468,307274,319565,323662},
{client_msstate,msg_store_transient,
<<32,49,130,217,86,65,129,214,32,140,24,34,
254,129,171,73>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[]}}},
{state,290886,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
<5297.261.0>,294983,286789,299080,303177}},
true,0,0,0,infinity,0,0,0,0,0,
{rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
[],
{set,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}}},
{state,
{queue,[],[],0},
{active,1406466722448182,1.0}},
undefined,undefined,undefined,undefined,
{state,fine,5000,undefined},
{0,nil},
undefined,undefined,undefined,
{state,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
delegate},
{resource,<<"eventing">>,exchange,
<<"EnterpriseEventDeadLetter">>},
<<"SalesCenter2.BPClaims_ClosedClaim">>,
undefined,2,running}],
[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,wake_up,3,
[{file,"proc_lib.erl"},{line,249}]}]},
{gen_server2,call,
[<5297.332.0>,
{add_on_right,{20,<0.323.0>}},
infinity]}}}
** When Server state == {state,
{amqqueue,
{resource,<<"eventing">>,queue,
<<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
true,false,none,
[{<<"x-dead-letter-exchange">>,longstr,
<<"EnterpriseEventDeadLetter">>},
{<<"x-dead-letter-routing-key">>,longstr,
<<"Insight.BPClaims_ModifiedWorkNotAvailable">>}],
<5251.335.0>,
[<5297.331.0>],
[<5297.331.0>],
[{vhost,<<"eventing">>},
{name,<<"ha-all">>},
{pattern,<<"^.*">>},
{'apply-to',<<"queues">>},
{definition,
[{<<"ha-mode">>,<<"all">>},
{<<"ha-sync-mode">>,<<"automatic">>}]},
{priority,0}],
[{<5297.332.0>,<5297.331.0>},
{<5251.336.0>,<5251.335.0>}],
[]},
<0.323.0>,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
{0,nil},
{qistate,
"d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/queues/11YNGR2UMDIGSF4OWRUUJ2HUX",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
[]},
undefined,0,65536,
#Fun<rabbit_variable_queue.2.51560583>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<142,49,10,175,39,218,164,74,2,215,253,147,201,
168,141,2>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,303177,
"d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_persistent"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_persistent",
<0.251.0>,307274,299080,311371,315468},
{client_msstate,msg_store_transient,
<<18,152,170,252,55,149,243,37,252,234,131,180,53,
66,14,182>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,282692,
"d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_transient"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_transient",
<0.246.0>,286789,278595,290886,294983}},
true,0,0,0,infinity,0,0,0,0,0,
{rates,0.0,0.0,0.0,0.0,{1406,466811,43211}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0},
undefined,undefined,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{state,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
delegate},
undefined}
** Reason for termination ==
** {{function_clause,
[{rabbit_amqqueue_process,handle_call,
[{add_on_right,{20,<0.323.0>}},
{<0.323.0>,#Ref<0.0.0.2348>},
{q,{amqqueue,
{resource,<<"eventing">>,queue,
<<"SalesCenter2.BPClaims_ClosedClaim">>},
true,false,none,
[{<<"x-dead-letter-exchange">>,longstr,
<<"EnterpriseEventDeadLetter">>},
{<<"x-dead-letter-routing-key">>,longstr,
<<"SalesCenter2.BPClaims_ClosedClaim">>}],
<5297.332.0>,
[<5251.310.0>],
[<5251.310.0>],
[{vhost,<<"eventing">>},
{name,<<"ha-all">>},
{pattern,<<"^.*">>},
{'apply-to',<<"queues">>},
{definition,
[{<<"ha-mode">>,<<"all">>},
{<<"ha-sync-mode">>,<<"automatic">>}]},
{priority,0}],
[{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
[]},
none,false,rabbit_mirror_queue_master,
{state,
{resource,<<"eventing">>,queue,
<<"SalesCenter2.BPClaims_ClosedClaim">>},
<5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
{0,nil},
{qistate,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
[]},
undefined,0,65536,
#Fun<rabbit_variable_queue.2.51560583>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<26,131,31,121,222,176,41,144,168,192,42,159,
222,13,179,107>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{state,311371,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
<5297.266.0>,315468,307274,319565,323662},
{client_msstate,msg_store_transient,
<<32,49,130,217,86,65,129,214,32,140,24,34,254,
129,171,73>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{state,290886,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
rabbit_msg_store_ets_index,
"d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
<5297.261.0>,294983,286789,299080,303177}},
true,0,0,0,infinity,0,0,0,0,0,
{rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
[],
{set,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
{state,{queue,[],[],0},{active,1406466722448182,1.0}},
undefined,undefined,undefined,undefined,
{state,fine,5000,undefined},
{0,nil},
undefined,undefined,undefined,
{state,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
delegate},
{resource,<<"eventing">>,exchange,
<<"EnterpriseEventDeadLetter">>},
<<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
{gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}
This e-mail is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this e-mail in error, please tell us immediately by return e-mail and delete the document. No recipient may use the information in this e-mail in violation of any civil or criminal statute. Sentry disclaims all liability for any unauthorized uses of this e-mail or its contents. Sentry accepts no liability or responsibility for any damage caused by any virus transmitted with this e-mail.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140728/c09ad936/attachment.html>
More information about the rabbitmq-discuss
mailing list