[rabbitmq-discuss] .NET client: intermittent AlreadyClosedException - connected host has failed to respond

[Andrei Volkov]

Before you spend too much time on this, I think I found the
problem. Posting it here for the benefit of future strugglers.

It looks like a TCP port exhaustion on the client side.
What happens is, unlike our main app, our healthcheck page opens (and
closes) a new connection every time it is hit. The healthcheck page is hit
every 5 seconds. By default on Windows the ephemeral TCP port can't be
reused until after 240 seconds after it's closed. Hence the port exhaustion.

Cheers.

On Thu, Nov 29, 2012 at 11:37 AM, Andrei Volkov <zvolkov at gmail.com> wrote:

> Hi Emile!
>
> We do have the hearbeat enabled, and set to 5 seconds. According to traces
> it is actually happening every 2 to 4 seconds. Rackspace have confirmed the
> network hardware is configured to drop idle connections after one hour.
> This should not be an issue then.
>
> We also see a large number of connections opened and closed immediately.
> According to Rackspace, the connections are closed with an RST instead of
> an FIN, however I don't see the "abrupt" termination errors in RabbitMQ
> logs, all I see is a large number (about 35,000 per day) of messages like
> this:
>
> =INFO REPORT==== 21-Nov-2012::10:57:26 ===
> accepting AMQP connection <0.30152.855> (192.168.221.206:58724 ->
> 192.168.220.208:5672)
>
> =INFO REPORT==== 21-Nov-2012::10:57:26 ===
> closing AMQP connection <0.30152.855> (192.168.221.206:58724 ->
> 192.168.220.208:5672)
>
> Is this normal? What could be causing this? How can I further troubleshoot
> this?
>
> Many thanks!
>
> ~AV
>
>
> On Wed, Nov 28, 2012 at 5:18 AM, Emile Joubert <emile at rabbitmq.com> wrote:
>
>> Hi Andrei,
>>
>> On 27/11/12 17:32, Andrei Volkov wrote:
>>
>>> I think (but not sure) it seems to happen after a period of inactivity,
>>> after the connection has been open but idle for a while.
>>>
>>
>> In that case a possible cause is that the connection dropped due to a
>> time-out. You can work around this by enabling AMQP heartbeats. They will
>> prevent inactivity and detect broken connections sooner. See the client
>> library documentation for 2.8.7:
>>
>> http://www.rabbitmq.com/**releases/rabbitmq-dotnet-**
>> client/v2.8.7/rabbitmq-dotnet-**client-2.8.7-client-htmldoc/**
>> html/type-RabbitMQ.Client.**ConnectionFactory.html#field-**
>> F:RabbitMQ.Client.**ConnectionFactory.**RequestedHeartbeat<http://www.rabbitmq.com/releases/rabbitmq-dotnet-client/v2.8.7/rabbitmq-dotnet-client-2.8.7-client-htmldoc/html/type-RabbitMQ.Client.ConnectionFactory.html#field-F:RabbitMQ.Client.ConnectionFactory.RequestedHeartbeat>
>>
>> You can also diagnose connectivity issues by issuing "rabbitmqctl
>> list_connections" on the broker and by checking "netstat" on both the
>> broker and the client.
>>
>>
>>
>>
>> -Emile
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121129/54e8116e/attachment.htm>