[rabbitmq-discuss] Hanging connections/channels after crash

[Simon MacMullen]

Hi.

These connections / channels do not exist. You're seeing a bug in the 
management plugin where it will retain information about connections and 
channels that were alive on a cluster node when it crashed.

This bug was fixed in RabbitMQ 3.0.3.

Cheers, Simon

On 17/04/13 09:57, Dirk Plate wrote:
> Hello,
> we have a cluster of three RabbitMQs (3.0.1). After having too many unacked messages (because of an error on client side) we ran into a memory error ("Cannot allocate 1167696400 bytes of memory (of type "old_heap")".
> Now the management plugin shows "blocked" and "blocking" connections. Additional one of these connections has a channel with 1.6m unacked messages. These connections and channels are not displayed when calling "rabbitmqctl list_connections" or "rabbitmqctl list_channels"!
> These connections/channels survived the restart of the client server and the RabbitMQ server.
> Closing the connection with the management plugin results in this error:
>
> {throw,{error,{not_a_connection_pid,<0.449.0>}},       [{rabbit_networking,close_connection,2},        {rabbit_mgmt_wm_connection,delete_resource,2},        {webmachine_resource,resource_call,3},        {webmachine_resource,do,3},        {webmachine_decision_core,resource_call,1},        {webmachine_decision_core,decision,1},        {webmachine_decision_core,handle_request,2},        {rabbit_webmachine,'-makeloop/1-fun-0-',2}]}
>
> This is the ouput of the rest client for one of these connections (ip addresses censored):
>
>      {
>         "recv_oct": 636365,
>         "recv_cnt": 949,
>         "send_oct": 1696,
>         "send_cnt": 164,
>         "send_pend": 0,
>         "state": "blocked",
>         "last_blocked_by": "resource",
>         "last_blocked_age": 1.251632,
>         "channels": 5,
>         "recv_oct_details":
>         {
>             "rate": 0,
>             "interval": 32663327,
>             "last_event": 1365522007990
>         },
>         "send_oct_details":
>         {
>             "rate": 0,
>             "interval": 32663327,
>             "last_event": 1365522007990
>         },
>         "type": "network",
>         "node": "m12n-production at rabbitmq1",
>         "name": "###.##.1.102:13353 -> ###.##.1.70:5672",
>         "port": 5672,
>         "peer_port": 13353,
>         "host": "###.##.1.70",
>         "peer_host": "###.##.1.102",
>         "ssl": false,
>         "peer_cert_subject": "",
>         "peer_cert_issuer": "",
>         "peer_cert_validity": "",
>         "auth_mechanism": "PLAIN",
>         "ssl_protocol": "",
>         "ssl_key_exchange": "",
>         "ssl_cipher": "",
>         "ssl_hash": "",
>         "protocol": "AMQP 0-9-1",
>         "user": "m12n",
>         "vhost": "prod",
>         "timeout": 30,
>         "frame_max": 131072,
>         "client_properties":
>         {
>             "product": "RabbitMQ",
>             "information": "Licensed under the MPL. See http://www.rabbitmq.com/",
>             "platform": "Java",
>             "capabilities":
>             {
>                 "exchange_exchange_bindings": true,
>                 "consumer_cancel_notify": true,
>                 "basic.nack": true,
>                 "publisher_confirms": true
>             },
>             "copyright": "Copyright (C) 2007-2012 VMware, Inc.",
>             "version": "3.0.1"
>         }
>      }
>
> How can we kill these connections/channels?
>
> Greetings, Dirk
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>


-- 
Simon MacMullen
RabbitMQ, VMware