[rabbitmq-discuss] Cluster Node Crashes
Cordell, Ron
Ron.Cordell at RelayHealth.com
Tue Sep 17 23:48:24 BST 2013
We are running RabbitMQ 3.1.5 and Erlang 16B01 x64 on Windows Server 2008R2 in a clustered configuration.
We are seeing nodes crash at random and don't understand what's going on. However, there is a common theme in the logs, of which a sample is here:
Does anyone have any insight into what might be happening? We see this in several different environments with similar cluster configurations and all running the same RabbitMQ/Erlang versions…
=INFO REPORT==== 13-Sep-2013::15:10:10 ===
accepting AMQP connection <0.5121.11> (10.100.183.10:63072 -> 10.100.183.75:5672)
=INFO REPORT==== 13-Sep-2013::15:10:10 ===
accepting AMQP connection <0.5149.11> (10.100.183.10:63075 -> 10.100.183.75:5672)
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_sendtopracticefusionactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.416.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.409.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-message_tracking_control_sjpeweb01' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.532.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.529.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_lookupproviderinregistryactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.352.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.349.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-compensate_sendtocloverleafactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.356.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.357.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-subscriber_activity_service_control_sjpeweb02' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.376.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.373.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_fakefaultedactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.348.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.345.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-inbound_activity_service_control_sjpeweb04' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.388.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.385.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_mongopersisteractivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.320.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.317.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-compensate_sendtorouteractivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.344.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.341.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-compensate_preparedocumentforroutingactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.384.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.381.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_unrecognizedmessageactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.316.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.313.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-execute_sendtocloverleafactivity' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.360.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.353.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-inbound_activity_service_control_sjpeweb02' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.372.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.369.0>
=INFO REPORT==== 15-Sep-2013::06:34:15 ===
Mirrored-queue (queue 'ha-exchange_infrastructure_control_sjpeweb02' in vhost 'vortex'): Slave <rabbit at SJPERMQ05.2.304.0> saw deaths of mirrors <rabbit at SJPERMQ04.3.301.0>
=ERROR REPORT==== 15-Sep-2013::06:34:16 ===
** Generic server <0.336.0> terminating
** Last message in was {gm_deaths,[<5111.330.0>]}
** When Server state == {state,
{amqqueue,
{resource,<<"vortex">>,queue,
<<"ha-execute_sendtorouteractivity">>},
true,false,none,
[{<<"x-ha-policy">>,longstr,<<"all">>}],
<5111.329.0>,
[<5111.329.0>,<7010.329.0>,<7011.333.0>],
[<7011.333.0>,<7010.329.0>,<5111.329.0>],
[{vhost,<<"vortex">>},
{name,<<"ha-policy">>},
{pattern,<<"^ha-.*">>},
{definition,
[{<<"ha-mode">>,<<"all">>},
{<<"ha-sync-mode">>,<<"automatic">>}]},
{priority,0}],
[{<7011.334.0>,<7011.333.0>},
{<7010.330.0>,<7010.329.0>},
{<5111.330.0>,<5111.329.0>},
{<7009.328.0>,<7009.327.0>}]},
<0.337.0>,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
{0,nil},
{qistate,
"e:/RabbitMQ/db/rabbit-mnesia/queues/CZDXJN5ZJBZN98DWOHU2MZHL",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
[]},
undefined,0,65536,
#Fun<rabbit_variable_queue.2.81334491>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<127,33,92,106,192,40,110,10,76,157,95,226,1,93,
82,226>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,348236,
"e:/RabbitMQ/db/rabbit-mnesia/msg_store_persistent"},
rabbit_msg_store_ets_index,
"e:/RabbitMQ/db/rabbit-mnesia/msg_store_persistent",
<0.269.0>,352333,344139,356430,360527},
{client_msstate,msg_store_transient,
<<95,20,53,215,211,161,4,63,158,97,9,63,159,60,91,
221>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,327751,
"e:/RabbitMQ/db/rabbit-mnesia/msg_store_transient"},
rabbit_msg_store_ets_index,
"e:/RabbitMQ/db/rabbit-mnesia/msg_store_transient",
<0.264.0>,331848,323654,335945,340042}},
true,0,0,0,infinity,0,0,0,0,0,
{rates,
{{1379,109357,6220},0},
{{1379,109357,6220},0},
0.0,0.0,
{1379,109829,181271}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0,
{rates,
{{1379,109357,6220},0},
{{1379,109357,6220},0},
0.0,0.0,
{1379,109829,181271}}},
undefined,#Ref<0.0.55.58770>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
0}
** Reason for termination ==
** {{badmatch,{error,not_found}},
[{rabbit_amqqueue_process,i,2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,emit_stats,2,[]},
{rabbit_event,if_enabled,3,[]},
{rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},
{rabbit_amqqueue_process,terminate_shutdown,2,[]},
{gen_server2,terminate,3,[]}]}
** In 'terminate' callback with reason ==
** normal
=INFO REPORT==== 15-Sep-2013::06:34:16 ===
closing AMQP connection <0.4955.11> (10.100.183.10:62972 -> 10.100.183.75:5672)
=INFO REPORT==== 15-Sep-2013::06:34:16 ===
closing AMQP connection <0.3177.11> (10.100.183.10:49728 -> 10.100.183.75:5672)
=INFO REPORT==== 15-Sep-2013::06:34:25 ===
accepting AMQP connection <0.24239.16> (10.100.183.10:56699 -> 10.100.183.75:5672)
=INFO REPORT==== 15-Sep-2013::06:34:25 ===
accepting AMQP connection <0.24248.16> (10.100.183.10:55592 -> 10
And a little later in the same log, here:
=INFO REPORT==== 15-Sep-2013::06:34:25 ===
Synchronising queue 'ha-compensate_fakecompensationfaultedactivity' in vhost 'vortex': all slaves already synced
=INFO REPORT==== 15-Sep-2013::06:34:25 ===
Synchronising queue 'ha-compensate_fakecompensationfaultedactivity' in vhost 'vortex': 0 messages to synchronise
=INFO REPORT==== 15-Sep-2013::06:34:25 ===
Synchronising queue 'ha-compensate_fakecompensationfaultedactivity' in vhost 'vortex': all slaves already synced
=INFO REPORT==== 15-Sep-2013::06:34:25 ===
Synchronising queue 'ha-compensate_fakecompensationfaultedactivity' in vhost 'vortex': 0 messages to synchronise
=INFO REPORT==== 15-Sep-2013::06:34:25 ===
Synchronising queue 'ha-compensate_fakecompensationfaultedactivity' in vhost 'vortex': all slaves already synced
=INFO REPORT==== 15-Sep-2013::06:34:45 ===
accepting AMQP connection <0.24463.16> (10.100.183.10:55948 -> 10.100.183.75:5672)
=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
** Generic server <0.353.0> terminating
** Last message in was {'DOWN',#Ref<0.0.0.3812>,process,<5111.350.0>,
{function_clause,
[{orddict,fetch,
[{9,<5111.350.0>},[]],
[{file,"orddict.erl"},{line,72}]},
{gm,check_neighbours,1,[]},
{gm,handle_cast,2,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,wake_up,3,
[{file,"proc_lib.erl"},{line,249}]}]}}
** When Server state == {state,
{18,<0.353.0>},
{{22,<0.24486.16>},#Ref<0.0.55.61004>},
{{9,<5111.350.0>},#Ref<0.0.0.3812>},
{resource,<<"vortex">>,queue,
<<"ha-execute_lookupproviderinregistryactivity">>},
rabbit_mirror_queue_slave,
{24,
[{{9,<5111.350.0>},
{view_member,
{9,<5111.350.0>},
[],
{18,<0.353.0>},
{12,<7010.350.0>}}},
{{12,<7010.350.0>},
{view_member,
{12,<7010.350.0>},
[],
{9,<5111.350.0>},
{22,<0.24486.16>}}},
{{18,<0.353.0>},
{view_member,
{18,<0.353.0>},
[],
{22,<0.24486.16>},
{9,<5111.350.0>}}},
{{22,<0.24486.16>},
{view_member,
{22,<0.24486.16>},
[],
{12,<7010.350.0>},
{18,<0.353.0>}}}]},
0,
[{{9,<5111.350.0>},
{member,{[{2,process_death}],[]},2,1}},
{{12,<7010.350.0>},
{member,{[{2,process_death}],[]},2,1}},
{{18,<0.353.0>},{member,{[],[]},0,0}},
{{22,<0.24486.16>},{member,{[],[]},-1,-1}}],
[<0.352.0>],
{[],[]},
[],undefined,
#Fun<rabbit_misc.execute_mnesia_transaction.1>}
** Reason for termination ==
** {function_clause,[{orddict,fetch,
[{18,<0.353.0>},[]],
[{file,"orddict.erl"},{line,72}]},
{gm,check_neighbours,1,[]},
{gm,handle_info,2,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
** Generic server <0.24486.16> terminating
** Last message in was {'DOWN',#Ref<0.0.55.60998>,process,<0.353.0>,
{function_clause,
[{orddict,fetch,
[{18,<0.353.0>},[]],
[{file,"orddict.erl"},{line,72}]},
{gm,check_neighbours,1,[]},
{gm,handle_info,2,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,wake_up,3,
[{file,"proc_lib.erl"},{line,249}]}]}}
** When Server state == {state,
{22,<0.24486.16>},
{{12,<7010.350.0>},#Ref<0.0.55.60997>},
{{18,<0.353.0>},#Ref<0.0.55.60998>},
{resource,<<"vortex">>,queue,
<<"ha-execute_lookupproviderinregistryactivity">>},
rabbit_mirror_queue_coordinator,
{23,
[{{9,<5111.350.0>},
{view_member,
{9,<5111.350.0>},
[],
{18,<0.353.0>},
{12,<7010.350.0>}}},
{{12,<7010.350.0>},
{view_member,
{12,<7010.350.0>},
[],
{9,<5111.350.0>},
{22,<0.24486.16>}}},
{{18,<0.353.0>},
{view_member,
{18,<0.353.0>},
[{6,<7009.348.0>}],
{22,<0.24486.16>},
{9,<5111.350.0>}}},
{{22,<0.24486.16>},
{view_member,
{22,<0.24486.16>},
[],
{12,<7010.350.0>},
{18,<0.353.0>}}}]},
-1,
[{{6,<7009.348.0>},{member,{[],[]},15,15}},
{{9,<5111.350.0>},
{member,{[{2,process_death}],[]},2,1}},
{{12,<7010.350.0>},
{member,{[{2,process_death}],[]},2,1}},
{{18,<0.353.0>},{member,{[],[]},0,0}},
{{22,<0.24486.16>},{member,{[],[]},-1,-1}}],
[<0.24485.16>],
{[],[]},
[],undefined,
#Fun<rabbit_misc.execute_mnesia_transaction.1>}
** Reason for termination ==
** {function_clause,[{orddict,fetch,
[{22,<0.24486.16>},[]],
[{file,"orddict.erl"},{line,72}]},
{gm,check_neighbours,1,[]},
{gm,handle_info,2,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}
=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
** Generic server <0.352.0> terminating
** Last message in was {gm_deaths,[<0.24486.16>,<0.353.0>,<7010.350.0>,
<5111.350.0>]}
** When Server state == {state,
{amqqueue,
{resource,<<"vortex">>,queue,
<<"ha-execute_lookupproviderinregistryactivity">>},
true,false,none,
[{<<"x-ha-policy">>,longstr,<<"all">>}],
<0.24484.16>,
[<5111.349.0>,<7010.349.0>,<7011.349.0>],
[<7011.349.0>,<7010.349.0>,<5111.349.0>],
[{vhost,<<"vortex">>},
{name,<<"ha-policy">>},
{pattern,<<"^ha-.*">>},
{definition,
[{<<"ha-mode">>,<<"all">>},
{<<"ha-sync-mode">>,<<"automatic">>}]},
{priority,0}],
[{<7011.350.0>,<7011.349.0>},
{<7010.350.0>,<7010.349.0>},
{<5111.350.0>,<5111.349.0>},
{<7009.348.0>,<7009.347.0>}]},
<0.353.0>,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
{0,nil},
{qistate,
"e:/RabbitMQ/db/rabbit-mnesia/queues/5MD5HQ7PCUA3C2UP3Z9HUANOV",
{{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
[]},
undefined,0,65536,
#Fun<rabbit_variable_queue.2.81334491>,
{0,nil}},
{{client_msstate,msg_store_persistent,
<<139,43,92,149,89,56,222,175,208,46,119,138,91,
209,199,19>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,348236,
"e:/RabbitMQ/db/rabbit-mnesia/msg_store_persistent"},
rabbit_msg_store_ets_index,
"e:/RabbitMQ/db/rabbit-mnesia/msg_store_persistent",
<0.269.0>,352333,344139,356430,360527},
{client_msstate,msg_store_transient,
<<241,205,183,204,120,219,30,233,243,36,174,112,
168,245,105,99>>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
{state,327751,
"e:/RabbitMQ/db/rabbit-mnesia/msg_store_transient"},
rabbit_msg_store_ets_index,
"e:/RabbitMQ/db/rabbit-mnesia/msg_store_transient",
<0.264.0>,331848,323654,335945,340042}},
true,0,0,0,infinity,0,0,0,0,0,
{rates,
{{1379,109827,41270},0},
{{1379,109827,41270},0},
0.0,0.0,
{1379,252056,729281}},
{0,nil},
{0,nil},
{0,nil},
{0,nil},
0,0,
{rates,
{{1379,109827,41270},0},
{{1379,109827,41270},0},
0.0,0.0,
{1379,252056,729281}}},
undefined,#Ref<0.0.55.61023>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
0}
** Reason for termination ==
** {{badmatch,{error,not_found}},
[{rabbit_amqqueue_process,i,2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,emit_stats,2,[]},
{rabbit_event,if_enabled,3,[]},
{rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,[]},
{rabbit_amqqueue_process,terminate_shutdown,2,[]},
{gen_server2,terminate,3,[]}]}
** In 'terminate' callback with reason ==
** normal
=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
** Generic server <0.24484.16> terminating
** Last message in was {init,new}
** When Server state == {q,{amqqueue,
{resource,<<"vortex">>,queue,
<<"ha-execute_lookupproviderinregistryactivity">>},
true,false,none,
[{<<"x-ha-policy">>,longstr,<<"all">>}],
<0.24484.16>,[],[],
[{vhost,<<"vortex">>},
{name,<<"ha-policy">>},
{pattern,<<"^ha-.*">>},
{definition,
[{<<"ha-mode">>,<<"all">>},
{<<"ha-sync-mode">>,<<"automatic">>}]},
{priority,0}],
[]},
none,false,undefined,undefined,
{[],[]},
undefined,undefined,undefined,undefined,
{state,fine,5000,undefined},
{0,nil},
undefined,undefined,undefined,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]}}},
undefined,undefined,undefined,running}
** Reason for termination ==
** {{badmatch,{error,not_found}},
[{rabbit_amqqueue_process,i,2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,emit_stats,2,[]},
{rabbit_event,if_enabled,3,[]},
{rabbit_amqqueue_process,declare,3,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
** Generic server <0.24482.16> terminating
** Last message in was {'$gen_cast',
{method,
{'basic.consume',0,
<<"ha-execute_lookupproviderinregistryactivity">>,
<<>>,false,false,false,false,[]},
none,noflow}}
** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,2,<0.24463.16>,
<0.24480.16>,<0.24463.16>,
<<"10.100.183.10:55948 -> 10.100.183.75:5672">>,
{lstate,<0.24481.16>,true,false},
none,1,
{[],[]},
{user,<<"SJPERMQVORTEXWRITE">>,[],
rabbit_auth_backend_ldap,
{impl,
"CN=SJPERMQ VORTEXWRITE,OU=PERMQ,OU=PERFORMANCE,OU=Environments,OU=Servers,DC=RHB,DC=AD",
<<"5k!Kz_T86Xa.W~">>}},
<<"vortex">>,
<<"ha-execute_lookupproviderinregistryactivity">>,
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{set,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
{set,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[]}}},
<0.24461.16>,
{state,fine,5000,#Ref<0.0.55.60968>},
false,1,
{{0,nil},{0,nil}},
[],
[{<<"publisher_confirms">>,bool,true},
{<<"exchange_exchange_bindings">>,bool,true},
{<<"consumer_cancel_notify">>,bool,true},
{<<"basic.nack">>,bool,true}],
none}
** Reason for termination ==
** {{{{badmatch,{error,not_found}},
[{rabbit_amqqueue_process,i,2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,emit_stats,2,[]},
{rabbit_event,if_enabled,3,[]},
{rabbit_amqqueue_process,declare,3,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},
{gen_server2,call,
[<0.24484.16>,
{basic_consume,false,<0.24482.16>,<0.24481.16>,true,
<<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>,false,none,
{'basic.consume_ok',<<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>}},
infinity]}},
[{gen_server2,call,3,[]},
{rabbit_channel,'-handle_method/3-fun-1-',7,[]},
{rabbit_misc,with_exit_handler,2,[]},
{rabbit_channel,handle_method,3,[]},
{rabbit_channel,handle_cast,2,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
=ERROR REPORT==== 15-Sep-2013::06:34:45 ===
AMQP connection <0.24463.16> (running), channel 2 - error:
{{{{badmatch,{error,not_found}},
[{rabbit_amqqueue_process,i,2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,emit_stats,2,[]},
{rabbit_event,if_enabled,3,[]},
{rabbit_amqqueue_process,declare,3,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},
{gen_server2,call,
[<0.24484.16>,
{basic_consume,false,<0.24482.16>,<0.24481.16>,true,
<<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>,false,none,
{'basic.consume_ok',<<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>}},
infinity]}},
[{gen_server2,call,3,[]},
{rabbit_channel,'-handle_method/3-fun-1-',7,[]},
{rabbit_misc,with_exit_handler,2,[]},
{rabbit_channel,handle_method,3,[]},
{rabbit_channel,handle_cast,2,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
=WARNING REPORT==== 15-Sep-2013::06:34:45 ===
Non-AMQP exit reason '{{{{badmatch,{error,not_found}},
[{rabbit_amqqueue_process,i,2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,[]},
{rabbit_amqqueue_process,emit_stats,2,[]},
{rabbit_event,if_enabled,3,[]},
{rabbit_amqqueue_process,declare,3,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]},
{gen_server2,call,
[<0.24484.16>,
{basic_consume,false,<0.24482.16>,<0.24481.16>,
true,<<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>,
false,none,
{'basic.consume_ok',
<<"amq.ctag-sPY5Ih3rfTx5gxibnPoXkw">>}},
infinity]}},
[{gen_server2,call,3,[]},
{rabbit_channel,'-handle_method/3-fun-1-',7,[]},
{rabbit_misc,with_exit_handler,2,[]},
{rabbit_channel,handle_method,3,[]},
{rabbit_channel,handle_cast,2,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}'
=INFO REPORT==== 15-Sep-2013::06:34:45 ===
closing AMQP connection <0.24463.16> (10.100.183.10:55948 -> 10.100.183.75:5672)
=WARNING REPORT==== 15-Sep-2013::06:34:47 ===
memory resource limit alarm set on node rabbit at SJPERMQ04.
**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130917/145eeaca/attachment.htm>
More information about the rabbitmq-discuss
mailing list