[rabbitmq-discuss] Orphaned channels after connection close in Rabbit 3.1.4

[Paul Bowsher]

Hi,

After the upgrade to RabbitMQ 3.1.4 we're seeing a large number of 
linearly-increasing channels which seem to hang around after the connection 
is closed. This doesn't happen on every queue (even those processed by the 
same software), only on a particular queue. The orphaned channels leave 20 
messages (the prefetch count) unacked.

Symptoms:

- Initially, larger than expected consumer count on queue from our 
monitoring
- Stopping all expected consumers on that channel removes the expected 
number of consumers, leaving orphans (700+ at present)
- Each orphaned consumer's channel is reachable using Management tool
- Each connection for the channel is reachable, is in either a "flow" or 
"blocked" state with zero data flow. Timeout is set to 600s (count doesn't 
decrease after 10 minutes)
- Forcing a stuck connection closed through the management interface 
results in a 500:

> The server encountered an error while processing this request:
> {exit,{normal,{gen_server,call,
>                           [<0.16806.1347>,
>                            {shutdown,"Closed via management plugin"},
>                            infinity]}},
>       [{gen_server,call,3,[{file,"gen_server.erl"},{line,188}]},
>        {rabbit_mgmt_wm_connection,delete_resource,2,[]},
>        {webmachine_resource,resource_call,3,[]},
>        {webmachine_resource,do,3,[]},
>        {webmachine_decision_core,resource_call,1,[]},
>        {webmachine_decision_core,decision,1,[]},
>        {webmachine_decision_core,handle_request,2,[]},
>        {rabbit_webmachine,'-makeloop/1-fun-0-',2,[]}]}


- The connection closure does actually succeed, and the count drops. 
- Forcing a normal connection closed still works correctly.
- Doing a netstat on both the client and server shows that the port 
associated with the connection is indeed completely closed, and not in any 
sort of TIME_WAIT or FIN_WAIT state

This issue occurred over the weekend, but as it was unchecked it eventually 
exhausted the server of socket descriptors, and we had to restart RabbitMQ 
to recover. It's happened again over night. We can't see anything in our 
client or server logs that give any indication as to the cause of it. We 
upgraded from 3.1.0 so unfortunately are unable to determine which version 
is responsible for the bug, if indeed it is a bug.

If anyone has seen this behaviour or has any suggestions, please let us 
know. Also, if we can provide any debug data let us know.

Thanks,

Paul Bowsher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130814/e9807b5e/attachment.htm>