[rabbitmq-discuss] RabbitMQ .NET Client Can Hang with SSL Connections

[Jonathan Oliver]

Awesome guys! Thanks for the fix:
http://www.rabbitmq.com/release-notes/README-3.2.4.txt
http://hg.rabbitmq.com/rabbitmq-dotnet-client/rev/49cfdfd34e15


On Sunday, February 16, 2014 11:33:13 AM UTC-7, Jonathan Oliver wrote:
>
> During periods of degraded WAN performance and availability, e.g. lost 
> packets, increased latency, etc. the .NET RabbitMQ connection is destroyed 
> according to the requested heartbeat and works as intended.  Our client is 
> observes this and then attempts to reconnect periodically until a 
> connection is re-established with the broker.  We've noticed however, that 
> sometimes these reconnection attempts hang *indefinitely/forever* despite 
> a configured RequestedConnectionTimeout value of 30 seconds.
>
> After months of seeing this problem every few weeks because of various 
> network hiccups, we were finally able to capture a snapshot and then 
> perform an inspection of the memory dump using WinDbg.  Here's the stack 
> trace (which uses the production RabbitMQ .NET client v3.2.1):
>
> 000000001adde530 000007fef77ad01a 
> System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, 
> System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)
> 000000001adde5d0 000007fef77ace88 
> System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)
> 000000001adde650 000007fef777294c 
> System.Net.FixedSizeReader.ReadPacket(Byte[], Int32, Int32)
> 000000001adde6a0 000007fef77d3803 
> System.Net.Security.SslState.StartReceiveBlob(Byte[], 
> System.Net.AsyncProtocolRequest)
> 000000001adde6e0 000007fef77d1f09 
> System.Net.Security.SslState.StartSendBlob(Byte[], Int32, 
> System.Net.AsyncProtocolRequest)
> 000000001adde760 000007fef77d3d26 
> System.Net.Security.SslState.ProcessReceivedBlob(Byte[], Int32, 
> System.Net.AsyncProtocolRequest)
> 000000001adde7d0 000007fef77d3816 
> System.Net.Security.SslState.StartReceiveBlob(Byte[], 
> System.Net.AsyncProtocolRequest)
> 000000001adde810 000007fef77d1f09 
> System.Net.Security.SslState.StartSendBlob(Byte[], Int32, 
> System.Net.AsyncProtocolRequest)
> 000000001adde890 000007fef77d1b85 
> System.Net.Security.SslState.ForceAuthentication(Boolean, Byte[], 
> System.Net.AsyncProtocolRequest)
> 000000001adde900 000007fef77d1856 
> System.Net.Security.SslState.ProcessAuthentication(System.Net.LazyAsyncResult)
>
> *RabbitMQ.Client.Impl.SocketFrameHandler_0_9..ctor(RabbitMQ.Client.AmqpTcpEndpoint, 
> ObtainSocket, Int32)*
> 000000001addea00 000007fe9a28c5e1 
> RabbitMQ.Client.Framing.Impl.v0_9_1.ProtocolBase.CreateFrameHandler(RabbitMQ.Client.AmqpTcpEndpoint, 
> ObtainSocket, Int32)
> 000000001addea50 000007fe9a28c245 
> RabbitMQ.Client.ConnectionFactory.FollowRedirectChain(Int32, 
> System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,Int32>, 
> System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,System.Exception>, 
> RabbitMQ.Client.AmqpTcpEndpoint[] ByRef, RabbitMQ.Client.AmqpTcpEndpoint)
> 000000001addeb50 000007fe9a28c05f 
> RabbitMQ.Client.ConnectionFactory.CreateConnection(Int32, 
> System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,Int32>, 
> System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,System.Exception>, 
> RabbitMQ.Client.AmqpTcpEndpoint[])
> 000000001addebe0 000007fe9a28bf51 
> RabbitMQ.Client.ConnectionFactory.CreateConnection(Int32)
> 000000001addec70 000007fe9a28bd6f *** ERROR: Module load completed but 
> symbols could not be loaded for Accounting.UsageHost.dll
>
> If you follow the stack trace through the .NET client source code, you'll 
> notice that a few of the methods are missing.  These have been optimized 
> away by the compiler when using a "Release" build.
>
> I have highlighted exactly where the problem is above and if you follow 
> the source code, you'll find the problem right here:
> *http://hg.rabbitmq.com/rabbitmq-dotnet-client/file/e399c21792d7/projects/client/RabbitMQ.Client/src/client/api/SslHelper.cs#l97 
> <http://hg.rabbitmq.com/rabbitmq-dotnet-client/file/e399c21792d7/projects/client/RabbitMQ.Client/src/client/api/SslHelper.cs#l97>*
>
> The problem is that there is *no timeout* value configured for SSL 
> connections.  This means that the SSL connection attempt will hang 
> indefinitely.  And since the connection doesn't exist yet, there's no way 
> for any kind of heartbeat thread to tear things down.
>
> The .NET docs also state that the default read/write timeout is set to 
> Infinite by default<http://msdn.microsoft.com/en-us/library/system.net.security.sslstream.readtimeout(v=vs.110).aspx>
> .
>
> The simple fix is as follows:
> sslStream.ReadTimeout = connectionTimeout;
> sslStream.WriteTimeout=connectionTimout;
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140304/e199715e/attachment.html>