[rabbitmq-discuss] Orphaned channels after connection close in Rabbit 3.1.4

[Paul Bowsher]

Hi Matthias,

Thanks for the reply. When you say resource alarms, do you mean on the
server or from the clients? We don't have any issues whatsoever there.
We've been running the same workload, and this issue only occurs since the
upgrade to 3.1.4.

We've performed some further experimentation since - we've configured
clients to only connect to node B. This has resolved the problem entirely.
The queues are still mirrored but we have no "stuck" connections. It seems
that we have a sick rabbit on node A. How can we help diagnose this?

Thanks,

Paul

Paul Bowsher


On Wed, Aug 14, 2013 at 10:23 PM, Matthias Radestock
<matthias at rabbitmq.com>wrote:

> Paul,
>
> On 14/08/13 10:41, Paul Bowsher wrote:
>
>> After the upgrade to RabbitMQ 3.1.4 we're seeing a large number of
>> linearly-increasing channels which seem to hang around after the
>> connection is closed.
>>
>
> This is very unlikely to be related to the upgrade...
>
>  - Initially, larger than expected consumer count on queue from our
>> monitoring
>> - Stopping all expected consumers on that channel removes the expected
>> number of consumers, leaving orphans (700+ at present)
>> - Each orphaned consumer's channel is reachable using Management tool
>> - Each connection for the channel is reachable, is in either a "flow" or
>> "blocked" state with zero data flow. Timeout is set to 600s (count
>> doesn't decrease after 10 minutes)
>>
>
> The above is all consistent with connections being blocked due to resource
> alarms. Blocked connections do not get closed until they are unblocked.
> Check the server logs for alarm warnings.
>
>  - Forcing a stuck connection closed through the management interface
>> results in a 500:
>>
>>     The server encountered an error while processing this request:
>>     {exit,{normal,{gen_server,**call,
>>                                [<0.16806.1347>,
>>                                 {shutdown,"Closed via management plugin"},
>>                                 infinity]}},
>>            [{gen_server,call,3,[{file,"**gen_server.erl"},{line,188}]},
>>             {rabbit_mgmt_wm_connection,**delete_resource,2,[]},
>>             {webmachine_resource,resource_**call,3,[]},
>>             {webmachine_resource,do,3,[]},
>>             {webmachine_decision_core,**resource_call,1,[]},
>>             {webmachine_decision_core,**decision,1,[]},
>>             {webmachine_decision_core,**handle_request,2,[]},
>>             {rabbit_webmachine,'-makeloop/**1-fun-0-',2,[]}]}
>>
>
> That looks like a bug. Alas I cannot see anything obviously wrong in the
> code and I cannot reproduce it :( Is this easily reproducible for you?
>
> Regards,
>
> Matthias.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130815/1cea71c0/attachment.htm>