<div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>we are use rabbitmq in our application, two hours ago, one of our app server is blocked when try to connect to rabbitmq, after check rabbitmq server , we found one node's memory is over watermark, a few <span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; ">minutes later, this node is down. after restart this node, the whole cluster sames work fine, but i notice there's a lot of connection in blocking and blocked state from web management，but use `</span>rabbitmqctl list_connections pid name peer_address state<span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; ">` in all nodes shows there is no connection in blocking/blocked…so this really make me confuse: </span></div><div><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; "><span class="Apple-tab-span" style="white-space:pre"> </span>1. after one node of whole cluster over watermark, but other node is work fine, my application can't connect to rabbitmq cluster? ps: we use spring.amqp & spring-rabbit with version 1.1.0.RELEASE</span></div><div><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; "><span class="Apple-tab-span" style="white-space:pre"> </span>2. </span><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; ">node will down </span><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; ">for what reason </span><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; ">when over watermark?</span></div><div><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; "><span class="Apple-tab-span" style="white-space:pre"> </span>3. why after restart node, there is still blocking connection, but with rabbitmqctl they all in running state?</span></div><div><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; "><br></span></div><div><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; "><span class="Apple-tab-span" style="white-space:pre"> </span>here is some logs from my rabbitmq server:</span></div><div><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; "><br></span></div><div><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; "><span class="Apple-tab-span" style="white-space:pre"> =INFO REPORT==== 1-Mar-2013::19:36:21 ===
vm_memory_high_watermark clear. Memory used:1656590680 allowed:1658778419
=INFO REPORT==== 1-Mar-2013::19:36:21 ===
alarm_handler: {clear,{resource_limit,memory,rabbit@cos22}}</span></span></div><div><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; "><br></span></div><div><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; "><span class="Apple-tab-span" style="white-space:pre"> </span>when i try to close blocked connection from web management, it goes error:</span></div><div><span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; "><span class="Apple-tab-span" style="white-space:pre"> </span></span></div><div><span class="Apple-tab-span" style="white-space:pre"> =INFO REPORT==== 1-Mar-2013::20:55:24 ===
Closing connection <0.17197.115> because "Closed via management plugin"
=ERROR REPORT==== 1-Mar-2013::20:55:24 ===
webmachine error: path="/api/connections/10.64.13.200%3A45891%20-%3E%2010.64.12.226%3A5672"
{throw,
{error,{not_a_connection_pid,<0.17197.115>}},
[{rabbit_networking,close_connection,2,
[{file,"src/rabbit_networking.erl"},{line,317}]},
{rabbit_mgmt_wm_connection,delete_resource,2,
[{file,"rabbitmq-management/src/rabbit_mgmt_wm_connection.erl"},
{line,52}]},
{webmachine_resource,resource_call,3,
[{file,
"webmachine-wrapper/webmachine-git/src/webmachine_resource.erl"},
{line,169}]},
{webmachine_resource,do,3,
[{file,
"webmachine-wrapper/webmachine-git/src/webmachine_resource.erl"},
{line,128}]},
{webmachine_decision_core,resource_call,1,
[{file,
"webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl"},
{line,48}]},
{webmachine_decision_core,decision,1,
[{file,
"webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl"},
{line,416}]},
{webmachine_decision_core,handle_request,2,
[{file,
"webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl"},
{line,33}]},
{rabbit_webmachine,'-makeloop/1-fun-0-',3,
[{file,"rabbitmq-mochiweb/src/rabbit_webmachine.erl"},{line,75}]}]}</span></div><div><br></div><div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>use rabbitmqctl shows all in running state:</div><div><br></div><div><span style="font-size: 14px;"><div>rabbitmqctl list_connections pid name peer_address state</div><div>Listing connections ...</div><div><rabbit@cos23.1.1271.51> 10.64.13.197:57321 -> 10.64.12.225:5672 10.64.13.197 running</div><div><rabbit@cos23.1.1100.51> 10.64.13.196:57240 -> 10.64.12.225:5672 10.64.13.196 running</div><div><rabbit@cos23.1.1056.51> 10.64.12.196:58608 -> 10.64.12.225:5672 10.64.12.196 running</div><div><rabbit@cos23.1.1079.51> 10.64.11.235:48962 -> 10.64.12.225:5672 10.64.11.235 running</div><div><rabbit@cos23.1.1419.51> 10.64.13.228:49857 -> 10.64.12.225:5672 10.64.13.228 running</div><div><rabbit@cos23.1.1049.51> 10.64.11.193:36387 -> 10.64.12.225:5672 10.64.11.193 running</div><div><rabbit@cos23.1.1159.51> 10.64.10.123:52017 -> 10.64.12.225:5672 10.64.10.123 running</div><div><rabbit@cos23.1.26289.45> 10.64.12.247:38504 -> 10.64.12.225:5672 10.64.12.247 running</div><div><rabbit@cos23.1.1121.51> 10.64.10.29:51483 -> 10.64.12.225:5672 10.64.10.29 running</div><div><rabbit@cos23.1.1067.51> 10.64.11.234:50244 -> 10.64.12.225:5672 10.64.11.234 running</div><div><rabbit@cos23.1.1149.51> 10.64.11.178:33795 -> 10.64.12.225:5672 10.64.11.178 running</div><div><rabbit@cos23.1.1136.51> 10.64.10.28:39557 -> 10.64.12.225:5672 10.64.10.28 running</div><div><rabbit@cos23.1.1370.51> 10.64.13.233:38766 -> 10.64.12.225:5672 10.64.13.233 running</div><div><rabbit@cos23.1.1388.51> 10.64.13.229:50932 -> 10.64.12.225:5672 10.64.13.229 running</div><div><rabbit@cos23.1.1254.51> 10.64.13.241:49311 -> 10.64.12.225:5672 10.64.13.241 running</div><div><rabbit@cos23.1.1031.51> 10.64.11.195:39455 -> 10.64.12.225:5672 10.64.11.195 running</div><div><rabbit@cos23.1.1038.51> 10.64.10.27:58938 -> 10.64.12.225:5672 10.64.10.27 running</div><div><rabbit@cos23.1.1167.51> 10.64.13.240:37777 -> 10.64.12.225:5672 10.64.13.240 running</div><div><rabbit@cos23.1.1442.51> 10.64.10.130:37251 -> 10.64.12.225:5672 10.64.10.130 running</div><div><rabbit@cos22.3.2659.0> 10.64.13.200:54840 -> 10.64.12.226:5672 10.64.13.200 running</div><div>...done.</div></span></div><div><br></div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>and there is a connection with a lot of channel is in blocked state, but i can't find this connection by use rabbitctl list_connections:</div><div><br></div><div><table class="list" style="border-collapse: collapse; border-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(204, 204, 204); margin-bottom: 1em; font-family: Verdana, sans-serif; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;"><tbody><tr class="alt1"><td style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: 17px; font-family: Verdana, sans-serif; padding: 4px; vertical-align: top; min-width: 5em; width: auto; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-right-width: 1px; border-right-style: solid; border-right-color: rgb(204, 204, 204); background-image: -webkit-gradient(linear, 0% 0%, 0% 100%, from(rgb(240, 240, 240)), to(rgb(224, 224, 224))); background-position: initial initial; background-repeat: initial initial;">AMQP 0-9-1</td><td style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: 17px; font-family: Verdana, sans-serif; padding: 4px; vertical-align: top; min-width: 5em; width: auto; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-right-width: 1px; border-right-style: solid; border-right-color: rgb(204, 204, 204); background-image: -webkit-gradient(linear, 0% 0%, 0% 100%, from(rgb(240, 240, 240)), to(rgb(224, 224, 224))); background-position: initial initial; background-repeat: initial initial;"><a href="http://10.64.12.226:55672/#/connections/10.64.13.200%3A45891%20-%3E%2010.64.12.226%3A5672" style="display: block; width: 301px; color: black; font-weight: bold; text-decoration: initial;">10.64.13.200:45891 -> 10.64.12.226:5672</a></td><td style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: 17px; font-family: Verdana, sans-serif; padding: 4px; vertical-align: top; min-width: 5em; width: auto; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-right-width: 1px; border-right-style: solid; border-right-color: rgb(204, 204, 204); background-image: -webkit-gradient(linear, 0% 0%, 0% 100%, from(rgb(240, 240, 240)), to(rgb(224, 224, 224))); background-position: initial initial; background-repeat: initial initial;">rabbit@cos22</td><td style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: 17px; font-family: Verdana, sans-serif; padding: 4px; vertical-align: top; min-width: 5em; width: auto; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-right-width: 1px; border-right-style: solid; border-right-color: rgb(204, 204, 204); background-image: -webkit-gradient(linear, 0% 0%, 0% 100%, from(rgb(240, 240, 240)), to(rgb(224, 224, 224))); background-position: initial initial; background-repeat: initial initial;">0B/s<sub style="display: block; font-size: 0.8em; color: rgb(136, 136, 136);">(49.2MB total)</sub></td><td style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: 17px; font-family: Verdana, sans-serif; padding: 4px; vertical-align: top; min-width: 5em; width: auto; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-right-width: 1px; border-right-style: solid; border-right-color: rgb(204, 204, 204); background-image: -webkit-gradient(linear, 0% 0%, 0% 100%, from(rgb(240, 240, 240)), to(rgb(224, 224, 224))); background-position: initial initial; background-repeat: initial initial;">0B/s<sub style="display: block; font-size: 0.8em; color: rgb(136, 136, 136);">(2.4MB total)</sub></td><td style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: 17px; font-family: Verdana, sans-serif; padding: 4px; vertical-align: top; min-width: 5em; width: auto; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-right-width: 1px; border-right-style: solid; border-right-color: rgb(204, 204, 204); background-image: -webkit-gradient(linear, 0% 0%, 0% 100%, from(rgb(240, 240, 240)), to(rgb(224, 224, 224))); background-position: initial initial; background-repeat: initial initial;">0s</td><td style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: 17px; font-family: Verdana, sans-serif; padding: 4px; vertical-align: top; min-width: 5em; width: auto; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-right-width: 1px; border-right-style: solid; border-right-color: rgb(204, 204, 204); background-image: -webkit-gradient(linear, 0% 0%, 0% 100%, from(rgb(240, 240, 240)), to(rgb(224, 224, 224))); background-position: initial initial; background-repeat: initial initial;">60920</td></tr></tbody></table></div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>thanks a lot for any help and <span style="color: rgb(61, 61, 61); font-family: Arial, sans-serif; line-height: 18px; ">suggestion.</span></div><div><br></div><div>-- </div><div>xzhang84@gmail.com</div><div>已使用 <a href="http://www.sparrowmailapp.com/?sig">Sparrow</a></div><div><br></div></div>