Before you spend too much time on this, I think I found the problem.�Posting it here for the benefit of future strugglers.<div><br><div>It looks like a TCP port�exhaustion�on the client side.</div><div>What happens is, unlike our main app, our healthcheck page opens (and closes) a new connection every time it is hit. The healthcheck page is hit every 5 seconds. By default on Windows the ephemeral TCP port can't be reused until after 240 seconds after it's closed. Hence the port exhaustion.</div>
<div><br>Cheers.<br><br><div class="gmail_quote">On Thu, Nov 29, 2012 at 11:37 AM, Andrei Volkov <span dir="ltr"><<a href="mailto:zvolkov@gmail.com" target="_blank">zvolkov@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Emile!�<div><br></div><div>We do have the hearbeat enabled, and set to 5 seconds. According to traces it is�actually�happening every 2 to 4 seconds. Rackspace have confirmed the network hardware is configured to drop idle connections after one hour. This should not be an issue then.<div>
<br></div><div>We also see a large number of connections opened and closed immediately. According to Rackspace, the connections are closed with an RST instead of an FIN, however I don't see the "abrupt" termination errors in RabbitMQ logs, all I see is a large number (about 35,000 per day) of messages like this:</div>
<div><br></div><div><div>=INFO REPORT==== 21-Nov-2012::10:57:26 ===</div><div>accepting AMQP connection <0.30152.855> (<a href="http://192.168.221.206:58724" target="_blank">192.168.221.206:58724</a> -> <a href="http://192.168.220.208:5672" target="_blank">192.168.220.208:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 21-Nov-2012::10:57:26 ===</div><div>closing AMQP connection <0.30152.855> (<a href="http://192.168.221.206:58724" target="_blank">192.168.221.206:58724</a> -> <a href="http://192.168.220.208:5672" target="_blank">192.168.220.208:5672</a>)</div>
<div><br></div><div>Is this normal? What could be causing this? How can I further troubleshoot this?</div><div><br></div><div>Many thanks!</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>~AV</div></font></span><div>
<div class="h5"><div><br></div><br><div class="gmail_quote">On Wed, Nov 28, 2012 at 5:18 AM, Emile Joubert <span dir="ltr"><<a href="mailto:emile@rabbitmq.com" target="_blank">emile@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Andrei,<br>
<br>
On 27/11/12 17:32, Andrei Volkov wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I think (but not sure) it seems to happen after a period of inactivity,<br>
after the connection has been open but idle for a while.<br>
</blockquote>
<br>
In that case a possible cause is that the connection dropped due to a time-out. You can work around this by enabling AMQP heartbeats. They will prevent inactivity and detect broken connections sooner. See the client library documentation for 2.8.7:<br>
<br>
<a href="http://www.rabbitmq.com/releases/rabbitmq-dotnet-client/v2.8.7/rabbitmq-dotnet-client-2.8.7-client-htmldoc/html/type-RabbitMQ.Client.ConnectionFactory.html#field-F:RabbitMQ.Client.ConnectionFactory.RequestedHeartbeat" target="_blank">http://www.rabbitmq.com/<u></u>releases/rabbitmq-dotnet-<u></u>client/v2.8.7/rabbitmq-dotnet-<u></u>client-2.8.7-client-htmldoc/<u></u>html/type-RabbitMQ.Client.<u></u>ConnectionFactory.html#field-<u></u>F:RabbitMQ.Client.<u></u>ConnectionFactory.<u></u>RequestedHeartbeat</a><br>
<br>
You can also diagnose connectivity issues by issuing "rabbitmqctl list_connections" on the broker and by checking "netstat" on both the broker and the client.<span><font color="#888888"><br>
<br>
<br>
<br>
<br>
-Emile<br>
<br>
<br>
<br>
</font></span></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div></div>