[rabbitmq-discuss] rabbitmq connections blocking but memory is blow watermark

xzhang84 at gmail.com xzhang84 at gmail.com
Fri Mar 1 14:39:55 GMT 2013

we are use rabbitmq in our application,  two hours ago, one of our app server is blocked when try to connect to rabbitmq, after check rabbitmq server ,  we found one node's memory is over watermark, a few minutes later, this node is down. after restart this node, the whole cluster sames work fine, but i notice there's a lot of connection in blocking and blocked state from web management,but use  `rabbitmqctl list_connections pid name peer_address state` in all nodes shows there is no connection in blocking/blocked…so this really make me confuse:  
1.  after  one node of whole cluster over watermark, but other node is work fine, my application can't connect to rabbitmq cluster?  ps:  we use spring.amqp & spring-rabbit with version 1.1.0.RELEASE
2.  node will down for what reason when over watermark?
3.  why after restart node, there is still blocking connection, but with rabbitmqctl they all in running state?

here is some logs from my rabbitmq server:

=INFO REPORT==== 1-Mar-2013::19:36:21 === vm_memory_high_watermark clear. Memory used:1656590680 allowed:1658778419 =INFO REPORT==== 1-Mar-2013::19:36:21 === alarm_handler: {clear,{resource_limit,memory,rabbit at cos22}}

when i try to close blocked connection from web management, it goes error:

=INFO REPORT==== 1-Mar-2013::20:55:24 === Closing connection <0.17197.115> because "Closed via management plugin" =ERROR REPORT==== 1-Mar-2013::20:55:24 === webmachine error: path="/api/connections/" {throw, {error,{not_a_connection_pid,<0.17197.115>}}, [{rabbit_networking,close_connection,2, [{file,"src/rabbit_networking.erl"},{line,317}]}, {rabbit_mgmt_wm_connection,delete_resource,2, [{file,"rabbitmq-management/src/rabbit_mgmt_wm_connection.erl"}, {line,52}]}, {webmachine_resource,resource_call,3, [{file, "webmachine-wrapper/webmachine-git/src/webmachine_resource.erl"}, {line,169}]}, {webmachine_resource,do,3, [{file, "webmachine-wrapper/webmachine-git/src/webmachine_resource.erl"}, {line,128}]}, {webmachine_decision_core,resource_call,1, [{file, "webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl"}, {line,48}]}, {webmachine_decision_core,decision,1, [{file, "webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl"}, {line,416}]}, {webmachine_decision_core,handle_request,2, [{file, "webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl"}, {line,33}]}, {rabbit_webmachine,'-makeloop/1-fun-0-',3, [{file,"rabbitmq-mochiweb/src/rabbit_webmachine.erl"},{line,75}]}]}

use rabbitmqctl shows all in running state:

rabbitmqctl list_connections pid name peer_address state
Listing connections ...
<rabbit at cos23.1.1271.51> ->    running
<rabbit at cos23.1.1100.51> ->    running
<rabbit at cos23.1.1056.51> ->    running
<rabbit at cos23.1.1079.51> ->    running
<rabbit at cos23.1.1419.51> ->    running
<rabbit at cos23.1.1049.51> ->    running
<rabbit at cos23.1.1159.51> ->    running
<rabbit at cos23.1.26289.45> ->    running
<rabbit at cos23.1.1121.51> ->     running
<rabbit at cos23.1.1067.51> ->    running
<rabbit at cos23.1.1149.51> ->    running
<rabbit at cos23.1.1136.51> ->     running
<rabbit at cos23.1.1370.51> ->    running
<rabbit at cos23.1.1388.51> ->    running
<rabbit at cos23.1.1254.51> ->    running
<rabbit at cos23.1.1031.51> ->    running
<rabbit at cos23.1.1038.51> ->     running
<rabbit at cos23.1.1167.51> ->    running
<rabbit at cos23.1.1442.51> ->    running
<rabbit at cos22.3.2659.0> ->    running

and there is a connection with a lot of channel is in blocked state, but i can't find this connection by use rabbitctl list_connections:

AMQP 0-9-1 -> (
rabbit at cos22
0B/s(49.2MB total)
0B/s(2.4MB total)

thanks a lot for any help and suggestion.

xzhang84 at gmail.com
已使用 Sparrow (http://www.sparrowmailapp.com/?sig)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130301/da22c7ab/attachment.htm>

More information about the rabbitmq-discuss mailing list