[rabbitmq-discuss] Hanging connections/channels after crash

Wed Apr 17 09:57:35 BST 2013

Hello,
we have a cluster of three RabbitMQs (3.0.1). After having too many unacked messages (because of an error on client side) we ran into a memory error ("Cannot allocate 1167696400 bytes of memory (of type "old_heap")".
Now the management plugin shows "blocked" and "blocking" connections. Additional one of these connections has a channel with 1.6m unacked messages. These connections and channels are not displayed when calling "rabbitmqctl list_connections" or "rabbitmqctl list_channels"!
These connections/channels survived the restart of the client server and the RabbitMQ server.
Closing the connection with the management plugin results in this error:

{throw,{error,{not_a_connection_pid,<0.449.0>}},       [{rabbit_networking,close_connection,2},        {rabbit_mgmt_wm_connection,delete_resource,2},        {webmachine_resource,resource_call,3},        {webmachine_resource,do,3},        {webmachine_decision_core,resource_call,1},        {webmachine_decision_core,decision,1},        {webmachine_decision_core,handle_request,2},        {rabbit_webmachine,'-makeloop/1-fun-0-',2}]}

This is the ouput of the rest client for one of these connections (ip addresses censored):

    {
       "recv_oct": 636365,
       "recv_cnt": 949,
       "send_oct": 1696,
       "send_cnt": 164,
       "send_pend": 0,
       "state": "blocked",
       "last_blocked_by": "resource",
       "last_blocked_age": 1.251632,
       "channels": 5,
       "recv_oct_details":
       {
           "rate": 0,
           "interval": 32663327,
           "last_event": 1365522007990
       },
       "send_oct_details":
       {
           "rate": 0,
           "interval": 32663327,
           "last_event": 1365522007990
       },
       "type": "network",
       "node": "m12n-production at rabbitmq1",
       "name": "###.##.1.102:13353 -> ###.##.1.70:5672",
       "port": 5672,
       "peer_port": 13353,
       "host": "###.##.1.70",
       "peer_host": "###.##.1.102",
       "ssl": false,
       "peer_cert_subject": "",
       "peer_cert_issuer": "",
       "peer_cert_validity": "",
       "auth_mechanism": "PLAIN",
       "ssl_protocol": "",
       "ssl_key_exchange": "",
       "ssl_cipher": "",
       "ssl_hash": "",
       "protocol": "AMQP 0-9-1",
       "user": "m12n",
       "vhost": "prod",
       "timeout": 30,
       "frame_max": 131072,
       "client_properties":
       {
           "product": "RabbitMQ",
           "information": "Licensed under the MPL. See http://www.rabbitmq.com/",
           "platform": "Java",
           "capabilities":
           {
               "exchange_exchange_bindings": true,
               "consumer_cancel_notify": true,
               "basic.nack": true,
               "publisher_confirms": true
           },
           "copyright": "Copyright (C) 2007-2012 VMware, Inc.",
           "version": "3.0.1"
       }
    }

How can we kill these connections/channels?

Greetings, Dirk