<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
I�m seeing a strange issue with the Rabbit cluster when we shut down the applications that connect to the cluster.</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
Here is the setup:</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
Rabbit 3.1.5 Erlang 16B02 running as a 2-node cluster behind an F5 load balancer</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
The F5 is currently set to send connections to ONLY the 01 node, but provides a virtual IP to the application.</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
The application consists of a 5 Windows services running on 4 servers each � all 5 services connect to the Rabbit cluster and those services are duplicated across 4 servers.</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
What seems to happen (but not always) is the following sequence of events:</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<ol style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<li>Issue commands to shut down Windows Services, Windows Services begin shutting down</li><li>At some point during the shutdown process we see the following in the Rabbit logs (truncated for brevity) - there is nothing in the logs before this event, by the way</li></ol>
<blockquote style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; margin: 0px 0px 0px 40px; border: none; padding: 0px;">
<blockquote style="margin:0 0 0 40px; border:none; padding:0px;">
<div>=WARNING REPORT==== 31-Oct-2013::13:35:57 ===</div>
<div>closing AMQP connection <0.3379.0> (10.100.193.10:50234 -> 10.100.193.80:5672):</div>
<div>connection_closed_abruptly</div>
<div><br>
</div>
<div>=WARNING REPORT==== 31-Oct-2013::13:35:57 ===</div>
<div>closing AMQP connection <0.3383.0> (10.100.193.10:50235 -> 10.100.193.80:5672):</div>
<div>connection_closed_abruptly</div>
<div><br>
</div>
<div>=WARNING REPORT==== 31-Oct-2013::13:35:57 ===</div>
<div>closing AMQP connection <0.3926.0> (10.100.193.10:50264 -> 10.100.193.80:5672):</div>
<div>connection_closed_abruptly</div>
</blockquote>
</blockquote>
<ol style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<li>The Windows Services that are trying to shut down hang, unable to shut down cleanly</li><li>The F5 load balancer is using a monitor to check to see if the Rabbit nodes are up via the HTTP API, and it shows that both are not responding, so the F5 stops routing traffic to either Rabbit node in the cluster</li><li>Logging onto the rabbit node,
<ol>
<li>we are able to use the Erlang ping/pong to verify connectivity between nodes</li><li>The management plugin UI is unresponsive</li><li>We issue a rabbitmqctl stop_app command to node 01, which responds, and we see the following in the rabbit log file:</li></ol>
</li></ol>
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">=INFO REPORT==== 31-Oct-2013::13:56:58 ===</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">Stopping RabbitMQ</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"><br>
</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">=ERROR REPORT==== 31-Oct-2013::13:57:00 ===</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** Generic server <0.6855.0> terminating</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** Last message in was {'$gen_cast',</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {method,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {'exchange.unbind',0,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"ha-execute_suppressmessagedeliveryactivity">>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"MassTransit.Diagnostics.Tracing:GetMessageTraceList">>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<>>,false,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> none,noflow}}</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,1,<0.4073.0>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <0.6853.0>,<0.4073.0>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"10.100.193.10:50286 -> 10.100.193.80:5672">>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {lstate,<0.6854.0>,false,false},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> none,1,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {user,<<"SJINRMQVORTEXWRITE">>,[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> rabbit_auth_backend_ldap,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {impl,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> "CN=SJINRMQ VORTEXWRITE,OU=INRMQ,OU=INTEGRATION,OU=Environments,OU=Servers,DC=RHB,DC=AD",</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"sQ*4~j5Ca9F_7G">>}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"vortex">>,<<>>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {dict,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {dict,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {dict,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {set,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {dict,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {set,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <0.4070.0>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {state,fine,5000,undefined},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> false,1,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{0,nil},{0,nil}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [{<<"publisher_confirms">>,bool,true},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {<<"exchange_exchange_bindings">>,bool,true},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {<<"consumer_cancel_notify">>,bool,true},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {<<"basic.nack">>,bool,true}],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> none}</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** Reason for termination == </font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** {{shutdown,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {gen_server,call,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [rabbit_auth_backend_ldap,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {check_resource,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [{username,<<"SJINRMQVORTEXWRITE">>},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {user_dn,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> "CN=SJINRMQ VORTEXWRITE,OU=INRMQ,OU=INTEGRATION,OU=Environments,OU=Servers,DC=RHB,DC=AD"},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {vhost,<<"vortex">>},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {resource,exchange},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {name,<<"ha-execute_suppressmessagedeliveryactivity">>},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {permission,write}],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {user,<<"SJINRMQVORTEXWRITE">>,[],rabbit_auth_backend_ldap,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {impl,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> "CN=SJINRMQ VORTEXWRITE,OU=INRMQ,OU=INTEGRATION,OU=Environments,OU=Servers,DC=RHB,DC=AD",</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"sQ*4~j5Ca9F_7G">>}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> infinity]}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [{gen_server,call,3,[{file,"gen_server.erl"},{line,188}]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {rabbit_access_control,check_access,5,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {rabbit_channel,check_resource_access,3,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {rabbit_channel,binding_action,9,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {rabbit_channel,handle_cast,2,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {gen_server2,handle_msg,2,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"><br>
</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">=ERROR REPORT==== 31-Oct-2013::13:57:00 ===</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** Generic server <0.16344.0> terminating</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** Last message in was {'$gen_cast',</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {method,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {'basic.publish',0,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"RelayHealth.MessageExchange.Core.Configuration.Events:MessageSubscribersLoaded">>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<>>,false,false},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {content,60,none,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<48,128,0,0,0,50,12,67,111,110,116,101,110,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> 116,45,84,121,112,101,83,0,0,0,32,97,112,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> 112,108,105,99,97,116,105,111,110,47,118,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> 110,100,46,109,97,115,115,116,114,97,110,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> 115,105,116,43,106,115,111,110,2,36,49,97,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> 102,100,48,48,48,48,45,53,50,97,102,45,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> 100,52,97,101,45,50,56,50,57,45,48,56,100,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> 48,97,48,52,56,50,53,102,102>>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> rabbit_framing_amqp_0_9_1,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [<<"{\r\n \"destinationAddress\": \"rabbitmq://rabbit.integration.rhb.ad/vortex/RelayHealth.MessageExchange.Core.Configuration.Events:MessageSubscribersLoaded\",\r\n
\"headers\": {},\r\n \"message\": {\r\n \"eventId\": \"1afd0000-52af-d4ae-dcd2-08d0a04825fe\",\r\n \"timeCompleted\": \"2013-10-31T08:56:47.178316Z\",\r\n \"duration\": \"00:00:00.0624004\",\r\n \"initiatingEventId\": \"1afd0000-52af-d4ae-6927-08d0a04825fc\",\r\n
\"machineName\": \"SJINWEB03\",\r\n \"processName\": \"RelayHealth.Services.SubscriberActivityService\",\r\n \"messageSubscriberCount\": 264\r\n },\r\n \"messageType\": [\r\n \"urn:message:RelayHealth.MessageExchange.Core.Configuration.MessageSubscribers:MessageSubscriberCache+MessageSubscribersLoadedImpl\",\r\n
\"urn:message:RelayHealth.MessageExchange.Core.Configuration.Events:MessageSubscribersLoaded\"\r\n ],\r\n \"sourceAddress\": \"rabbitmq://rabbit.integration.rhb.ad/vortex/ha-subscriber_activity_service_control_sjinweb03?ha=true&prefetch=1\"\r\n}">>]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> flow}}</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,1,<0.16335.0>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <0.16342.0>,<0.16335.0>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"10.100.193.10:54839 -> 10.100.193.80:5672">>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {lstate,<0.16343.0>,false,false},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> none,1,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {user,<<"SJINRMQVORTEXWRITE">>,[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> rabbit_auth_backend_ldap,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {impl,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> "CN=SJINRMQ VORTEXWRITE,OU=INRMQ,OU=INTEGRATION,OU=Environments,OU=Servers,DC=RHB,DC=AD",</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"sQ*4~j5Ca9F_7G">>}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"vortex">>,<<>>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {dict,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {dict,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {dict,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {set,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {dict,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {set,0,16,16,8,80,48,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> []},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],[]}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <0.16333.0>,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {state,fine,5000,undefined},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> true,233,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {{0,nil},{0,nil}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [{<<"publisher_confirms">>,bool,true},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {<<"exchange_exchange_bindings">>,bool,true},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {<<"consumer_cancel_notify">>,bool,true},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {<<"basic.nack">>,bool,true}],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> none}</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** Reason for termination == </font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** {{shutdown,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {gen_server,call,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [rabbit_auth_backend_ldap,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {check_resource,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [{username,<<"SJINRMQVORTEXWRITE">>},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {user_dn,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> "CN=SJINRMQ VORTEXWRITE,OU=INRMQ,OU=INTEGRATION,OU=Environments,OU=Servers,DC=RHB,DC=AD"},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {vhost,<<"vortex">>},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {resource,exchange},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {name,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"RelayHealth.MessageExchange.Core.Configuration.Events:MessageSubscribersLoaded">>},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {permission,write}],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {user,<<"SJINRMQVORTEXWRITE">>,[],rabbit_auth_backend_ldap,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {impl,</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> "CN=SJINRMQ VORTEXWRITE,OU=INRMQ,OU=INTEGRATION,OU=Environments,OU=Servers,DC=RHB,DC=AD",</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> <<"sQ*4~j5Ca9F_7G">>}}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> infinity]}},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> [{gen_server,call,3,[{file,"gen_server.erl"},{line,188}]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {rabbit_access_control,check_access,5,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {rabbit_channel,check_resource_access,3,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {rabbit_channel,handle_method,3,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {rabbit_channel,handle_cast,2,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {gen_server2,handle_msg,2,[]},</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"><br>
</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">=ERROR REPORT==== 31-Oct-2013::13:57:00 ===</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;">** Generic server <0.7275.0> terminating</font></div>
<div><font face="Courier"><span style="font-size: 12px;">** Last message in was {'$gen_cast�,</span></font></div>
<div style="color: rgb(0, 0, 0);"><font face="Courier" style="font-size: 12px;"><br>
</font></div>
<div>. . . Omitting the rest for brevity as there is one of these for every exchange</div>
</blockquote>
</blockquote>
</blockquote>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<ol style="color: rgb(0, 0, 0);">
<li>We also issue the stop-app on Rabbit node 02 of the cluster</li><li>Then we issue rabbitmqctl start_app on Node 01, and rabbit starts. The F5 sees that the node is available, adds it to the pool of available nodes and begins routing traffic to it, and the Windows services are able to complete shut down.</li></ol>
<div>I�ve verified that the application code is shutting down connections cleanly. We also don�t see this behavior with the same application in environments where no rabbit cluster is setup, just a single instance.</div>
<div>This appears to be something that happens to Rabbit itself � the Erlang portion of things seems to be working OK.</div>
<div><br>
</div>
<div>I�m attempting to replicate this behavior in other environments so I can try to isolate it but haven�t had a chance to do so thus far, but it does happen in every environment where we have a rabbit cluster.</div>
<div><br>
</div>
<div>Any ideas of what I might be missing?</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Ron</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<br>
</div>
</body>
</html>