<div dir="ltr">Hi Matthias,<div><br></div><div>Thanks for the reply. When you say resource alarms, do you mean on the server or from the clients? We don't have any issues whatsoever there. We've been running the same workload, and this issue only occurs since the upgrade to 3.1.4.</div>
<div><br></div><div>We've performed some further experimentation since - we've configured clients to only connect to node B. This has resolved the problem entirely. The queues are still mirrored but we have no "stuck" connections. It seems that we have a sick rabbit on node A. How can we help diagnose this?</div>
<div><br></div><div>Thanks,</div><div><br></div><div>Paul</div></div><div class="gmail_extra"><br clear="all"><div>Paul Bowsher</div>
<br><br><div class="gmail_quote">On Wed, Aug 14, 2013 at 10:23 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Paul,<br>
<br>
On 14/08/13 10:41, Paul Bowsher wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
After the upgrade to RabbitMQ 3.1.4 we're seeing a large number of<br>
linearly-increasing channels which seem to hang around after the<br>
connection is closed.<br>
</blockquote>
<br>
This is very unlikely to be related to the upgrade...<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- Initially, larger than expected consumer count on queue from our<br>
monitoring<br>
- Stopping all expected consumers on that channel removes the expected<br>
number of consumers, leaving orphans (700+ at present)<br>
- Each orphaned consumer's channel is reachable using Management tool<br>
- Each connection for the channel is reachable, is in either a "flow" or<br>
"blocked" state with zero data flow. Timeout is set to 600s (count<br>
doesn't decrease after 10 minutes)<br>
</blockquote>
<br>
The above is all consistent with connections being blocked due to resource alarms. Blocked connections do not get closed until they are unblocked. Check the server logs for alarm warnings.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- Forcing a stuck connection closed through the management interface<br>
results in a 500:<br>
<br>
� � The server encountered an error while processing this request:<br>
� � {exit,{normal,{gen_server,<u></u>call,<br>
� � � � � � � � � � � � � � � �[<0.16806.1347>,<br>
� � � � � � � � � � � � � � � � {shutdown,"Closed via management plugin"},<br>
� � � � � � � � � � � � � � � � infinity]}},<br>
� � � � � �[{gen_server,call,3,[{file,"<u></u>gen_server.erl"},{line,188}]},<br>
� � � � � � {rabbit_mgmt_wm_connection,<u></u>delete_resource,2,[]},<br>
� � � � � � {webmachine_resource,resource_<u></u>call,3,[]},<br>
� � � � � � {webmachine_resource,do,3,[]},<br>
� � � � � � {webmachine_decision_core,<u></u>resource_call,1,[]},<br>
� � � � � � {webmachine_decision_core,<u></u>decision,1,[]},<br>
� � � � � � {webmachine_decision_core,<u></u>handle_request,2,[]},<br>
� � � � � � {rabbit_webmachine,'-makeloop/<u></u>1-fun-0-',2,[]}]}<br>
</blockquote>
<br>
That looks like a bug. Alas I cannot see anything obviously wrong in the code and I cannot reproduce it :( Is this easily reproducible for you?<br>
<br>
Regards,<br>
<br>
Matthias.<br>
</blockquote></div><br></div>