[rabbitmq-discuss] Loss of durable queues on HA sync
Simon MacMullen
simon at rabbitmq.com
Mon Jul 28 19:05:14 BST 2014
Thanks for this. This is not a known issue, and not something we've seen
before.
In all cases it looks like a queue process crashed because it received a
message intended for a GM process. That should never be able to happen.
Could you send me full logs from all nodes around the time of the crash?
It would be useful to see in what order things happened, if anything led
up to this, when nodes restarted for autoheal (if any autoheal
happened), etc.
Cheers, Simon
On 28/07/14 17:40, Miller Andrew wrote:
> During some server reboots at our organization over the weekend, several
> nodes in our non-Prod cluster had some crashes, and in the end, 7 of 84
> durable queues in the cluster were gone. All nodes are running RabbitMQ
> 3.3.3 on Erlang R16B03-1. This is a 3-node cluster, with
> cluster_partition_handling = autoheal. I'm curious if this is a known
> issue with autoheal? I tried recreating the issue this morning by
> recreating the series of events, but was unable to. Our other 2-node
> non-Prod cluster, with no cluster_partition_handling, was also
> rebootted, but didn't have any problems.
>
> Thanks,
>
> Andrew Miller
>
> 08:12:01 - Node DIX-D-EVGAPP-02 (a disk node) saw node STR-D-EVGAPP-02
> (a memory node) go down (which had been the master node for many
> queues), and was promoted to master for 75 queues, including those which
> later would disappear.
>
> 08:13:30 - Node DIX-D-EVGAPP-02 saw node STR-D-EVGAPP-02 come back, and
> began synchronizing back. Most queues synchronized fine, but 7 crashed
> with errors similar to the following - and each of these queues were
> gone after this. The third node (another disk node) doesn't have any
> crashes in its log around this time.
>
> =ERROR REPORT==== 27-Jul-2014::08:13:30 ===
>
> ** Generic server <0.332.0> terminating
>
> ** Last message in was {add_on_right,{20,<4967.323.0>}}
>
> ** When Server state == {q,
>
> {amqqueue,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
> <0.332.0>,
>
> [<8174.310.0>],
>
> [<8174.310.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
>
> [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
>
> []},
>
> none,false,rabbit_mirror_queue_master,
>
> {state,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> <0.333.0>,<0.4223.0>,rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,222,
>
> 13,179,107>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
> <0.266.0>,315468,307274,319565,323662},
>
> {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
> 129,171,73>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
> <0.261.0>,294983,286789,299080,303177}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> [],
>
> {set,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}}},
>
>
> {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
> undefined,undefined,undefined,undefined,
>
> {state,fine,5000,undefined},
>
> {0,nil},
>
> undefined,undefined,undefined,
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> delegate},
>
> {resource,<<"eventing">>,exchange,
>
> <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,
>
> running}
>
> ** Reason for termination ==
>
> ** {function_clause,
>
> [{rabbit_amqqueue_process,handle_call,
>
> [{add_on_right,{20,<4967.323.0>}},
>
> {<4967.323.0>,#Ref<4967.0.0.2348>},
>
> {q,{amqqueue,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
> <0.332.0>,
>
> [<8174.310.0>],
>
> [<8174.310.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
> [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
>
> []},
>
> none,false,rabbit_mirror_queue_master,
>
> {state,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> <0.333.0>,<0.4223.0>,rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,222,
>
> 13,179,107>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
> <0.266.0>,315468,307274,319565,323662},
>
> {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
> 129,171,73>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
> <0.261.0>,294983,286789,299080,303177}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> [],
>
> {set,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
> {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
> undefined,undefined,undefined,undefined,
>
> {state,fine,5000,undefined},
>
> {0,nil},
>
> undefined,undefined,undefined,
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> delegate},
>
> {resource,<<"eventing">>,exchange,
>
> <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
> []},
>
> {gen_server2,handle_msg,2,[]},
>
> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
>
> 08:13:30 - Node STR-D-EVGAPP-02 also recorded crashes involving the same
> queues:
>
> =ERROR REPORT==== 27-Jul-2014::08:13:30 ===
>
> ** Generic server <0.323.0> terminating
>
> ** Last message in was {'$gen_cast',join}
>
> ** When Server state == {state,
>
> {20,<0.323.0>},
>
> {{20,<0.323.0>},undefined},
>
> {{20,<0.323.0>},undefined},
>
> {resource,<<"eventing">>,queue,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
>
>
> rabbit_mirror_queue_slave,undefined,-1,undefined,
>
> [<0.322.0>],
>
> {[],[]},
>
> [],0,undefined,
>
> #Fun<rabbit_misc.execute_mnesia_transaction.1>}
>
> ** Reason for termination ==
>
> ** {{function_clause,
>
> [{rabbit_amqqueue_process,handle_call,
>
> [{add_on_right,{20,<0.323.0>}},
>
> {<0.323.0>,#Ref<0.0.0.2348>},
>
> {q,{amqqueue,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
> <5297.332.0>,
>
> [<5251.310.0>],
>
> [<5251.310.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
>
> [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
>
> []},
>
> none,false,rabbit_mirror_queue_master,
>
> {state,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,
>
> 222,13,179,107>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
> <5297.266.0>,315468,307274,319565,323662},
>
> {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
> 129,171,73>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
> <5297.261.0>,294983,286789,299080,303177}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> [],
>
> {set,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
> {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
> undefined,undefined,undefined,undefined,
>
> {state,fine,5000,undefined},
>
> {0,nil},
>
> undefined,undefined,undefined,
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> delegate},
>
> {resource,<<"eventing">>,exchange,
>
> <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
> []},
>
> {gen_server2,handle_msg,2,[]},
>
> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
>
>
> {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}
>
> 08:13:31 - And again from node STR-D-EVGAPP-02:
>
> =ERROR REPORT==== 27-Jul-2014::08:13:31 ===
>
> ** Generic server <0.322.0> terminating
>
> ** Last message in was {'EXIT',<0.323.0>,
>
> {{function_clause,
>
> [{rabbit_amqqueue_process,handle_call,
>
> [{add_on_right,{20,<0.323.0>}},
>
> {<0.323.0>,#Ref<0.0.0.2348>},
>
> {q,
>
> {amqqueue,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
> <5297.332.0>,
>
> [<5251.310.0>],
>
> [<5251.310.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
> [{<5251.311.0>,<5251.310.0>},
>
> {<5297.333.0>,<5297.332.0>}],
>
> []},
>
> none,false,rabbit_mirror_queue_master,
>
> {state,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> <5297.333.0>,<5297.4223.0>,
>
> rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[],[]}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,
>
> 159,222,13,179,107>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[],[]}}},
>
> {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>
> <5297.266.0>,315468,307274,319565,323662},
>
> {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,
>
> 254,129,171,73>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[],[]}}},
>
> {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>
> <5297.261.0>,294983,286789,299080,303177}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> [],
>
> {set,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}}},
>
> {state,
>
> {queue,[],[],0},
>
> {active,1406466722448182,1.0}},
>
> undefined,undefined,undefined,undefined,
>
> {state,fine,5000,undefined},
>
> {0,nil},
>
> undefined,undefined,undefined,
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> delegate},
>
> {resource,<<"eventing">>,exchange,
>
> <<"EnterpriseEventDeadLetter">>},
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,
>
> undefined,2,running}],
>
> []},
>
> {gen_server2,handle_msg,2,[]},
>
> {proc_lib,wake_up,3,
>
> [{file,"proc_lib.erl"},{line,249}]}]},
>
> {gen_server2,call,
>
> [<5297.332.0>,
>
> {add_on_right,{20,<0.323.0>}},
>
> infinity]}}}
>
> ** When Server state == {state,
>
> {amqqueue,
>
> {resource,<<"eventing">>,queue,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>}],
>
> <5251.335.0>,
>
> [<5297.331.0>],
>
> [<5297.331.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
> [{<5297.332.0>,<5297.331.0>},
>
> {<5251.336.0>,<5251.335.0>}],
>
> []},
>
> <0.323.0>,rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/queues/11YNGR2UMDIGSF4OWRUUJ2HUX",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<142,49,10,175,39,218,164,74,2,215,253,147,201,
>
> 168,141,2>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> {state,303177,
>
>
> "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_persistent",
>
> <0.251.0>,307274,299080,311371,315468},
>
> {client_msstate,msg_store_transient,
>
>
> <<18,152,170,252,55,149,243,37,252,234,131,180,53,
>
> 66,14,182>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> {state,282692,
>
>
> "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at STR-D-EVGAPP-02-mnesia/msg_store_transient",
>
> <0.246.0>,286789,278595,290886,294983}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
> {rates,0.0,0.0,0.0,0.0,{1406,466811,43211}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> undefined,undefined,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> delegate},
>
> undefined}
>
> ** Reason for termination ==
>
> ** {{function_clause,
>
> [{rabbit_amqqueue_process,handle_call,
>
> [{add_on_right,{20,<0.323.0>}},
>
> {<0.323.0>,#Ref<0.0.0.2348>},
>
> {q,{amqqueue,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
> <5297.332.0>,
>
> [<5251.310.0>],
>
> [<5251.310.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
>
> [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
>
> []},
>
> none,false,rabbit_mirror_queue_master,
>
> {state,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,
>
> 222,13,179,107>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
> <5297.266.0>,315468,307274,319565,323662},
>
> {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
> 129,171,73>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit at DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
> <5297.261.0>,294983,286789,299080,303177}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> [],
>
> {set,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
> {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
> undefined,undefined,undefined,undefined,
>
> {state,fine,5000,undefined},
>
> {0,nil},
>
> undefined,undefined,undefined,
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> delegate},
>
> {resource,<<"eventing">>,exchange,
>
> <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
> []},
>
> {gen_server2,handle_msg,2,[]},
>
> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
>
>
> {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}
>
> This e-mail is confidential. If you are not the intended recipient, you
> must not disclose or use the information contained in it. If you have
> received this e-mail in error, please tell us immediately by return
> e-mail and delete the document. No recipient may use the information in
> this e-mail in violation of any civil or criminal statute. Sentry
> disclaims all liability for any unauthorized uses of this e-mail or its
> contents. Sentry accepts no liability or responsibility for any damage
> caused by any virus transmitted with this e-mail.
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
> please subscribe to the new list!
>
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
More information about the rabbitmq-discuss
mailing list