<div dir="ltr"><div>During periods of degraded WAN performance and availability, e.g. lost packets, increased latency, etc. the .NET RabbitMQ connection is destroyed according to the requested heartbeat and works as intended.  Our client is observes this and then attempts to reconnect periodically until a connection is re-established with the broker.  We've noticed however, that sometimes these reconnection attempts hang <b>indefinitely/forever</b> despite a configured RequestedConnectionTimeout value of 30 seconds.</div><div><br></div><div>After months of seeing this problem every few weeks because of various network hiccups, we were finally able to capture a snapshot and then perform an inspection of the memory dump using WinDbg.  Here's the stack trace (which uses the production RabbitMQ .NET client v3.2.1):</div><div><br></div><div><div>000000001adde530 000007fef77ad01a System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)</div><div>000000001adde5d0 000007fef77ace88 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)</div><div>000000001adde650 000007fef777294c System.Net.FixedSizeReader.ReadPacket(Byte[], Int32, Int32)</div><div>000000001adde6a0 000007fef77d3803 System.Net.Security.SslState.StartReceiveBlob(Byte[], System.Net.AsyncProtocolRequest)</div><div>000000001adde6e0 000007fef77d1f09 System.Net.Security.SslState.StartSendBlob(Byte[], Int32, System.Net.AsyncProtocolRequest)</div><div>000000001adde760 000007fef77d3d26 System.Net.Security.SslState.ProcessReceivedBlob(Byte[], Int32, System.Net.AsyncProtocolRequest)</div><div>000000001adde7d0 000007fef77d3816 System.Net.Security.SslState.StartReceiveBlob(Byte[], System.Net.AsyncProtocolRequest)</div><div>000000001adde810 000007fef77d1f09 System.Net.Security.SslState.StartSendBlob(Byte[], Int32, System.Net.AsyncProtocolRequest)</div><div>000000001adde890 000007fef77d1b85 System.Net.Security.SslState.ForceAuthentication(Boolean, Byte[], System.Net.AsyncProtocolRequest)</div><div>000000001adde900 000007fef77d1856 System.Net.Security.SslState.ProcessAuthentication(System.Net.LazyAsyncResult)</div><div><b>RabbitMQ.Client.Impl.SocketFrameHandler_0_9..ctor(RabbitMQ.Client.AmqpTcpEndpoint, ObtainSocket, Int32)<br></b></div><div>000000001addea00 000007fe9a28c5e1 RabbitMQ.Client.Framing.Impl.v0_9_1.ProtocolBase.CreateFrameHandler(RabbitMQ.Client.AmqpTcpEndpoint, ObtainSocket, Int32)</div><div>000000001addea50 000007fe9a28c245 RabbitMQ.Client.ConnectionFactory.FollowRedirectChain(Int32, System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,Int32>, System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,System.Exception>, RabbitMQ.Client.AmqpTcpEndpoint[] ByRef, RabbitMQ.Client.AmqpTcpEndpoint)</div><div>000000001addeb50 000007fe9a28c05f RabbitMQ.Client.ConnectionFactory.CreateConnection(Int32, System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,Int32>, System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,System.Exception>, RabbitMQ.Client.AmqpTcpEndpoint[])</div><div>000000001addebe0 000007fe9a28bf51 RabbitMQ.Client.ConnectionFactory.CreateConnection(Int32)</div><div>000000001addec70 000007fe9a28bd6f *** ERROR: Module load completed but symbols could not be loaded for Accounting.UsageHost.dll</div></div><div><br></div><div>If you follow the stack trace through the .NET client source code, you'll notice that a few of the methods are missing.  These have been optimized away by the compiler when using a "Release" build.</div><div><br></div><div>I have highlighted exactly where the problem is above and if you follow the source code, you'll find the problem right here:</div><div><b>http://hg.rabbitmq.com/rabbitmq-dotnet-client/file/e399c21792d7/projects/client/RabbitMQ.Client/src/client/api/SslHelper.cs#l97</b></div><div><br></div><div>The problem is that there is <b>no timeout</b> value configured for SSL connections.  This means that the SSL connection attempt will hang indefinitely.  And since the connection doesn't exist yet, there's no way for any kind of heartbeat thread to tear things down.</div><div><br></div><div>The .NET docs also state that the default read/write timeout is <a href="http://msdn.microsoft.com/en-us/library/system.net.security.sslstream.readtimeout(v=vs.110).aspx">set to Infinite by default</a>.</div><div><br></div><div>The simple fix is as follows:</div><div><span style="color: rgb(51, 51, 51); font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 23.799999237060547px;">sslStream.ReadTimeout = connectionTimeout;</span></div><div><span style="color: rgb(51, 51, 51); font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 23.799999237060547px;">sslStream.WriteTimeout=connectionTimout;</span></div><div><span style="color: rgb(51, 51, 51); font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 23.799999237060547px;"><br></span></div><div><span style="color: rgb(51, 51, 51); font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 23.799999237060547px;"><br></span></div><div><span style="color: rgb(51, 51, 51); font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 23.799999237060547px;"><br></span></div><div><br></div></div>