[rabbitmq-discuss] RabbitMQ .NET Client Can Hang with SSL Connections

[Jonathan Oliver]

During periods of degraded WAN performance and availability, e.g. lost 
packets, increased latency, etc. the .NET RabbitMQ connection is destroyed 
according to the requested heartbeat and works as intended.  Our client is 
observes this and then attempts to reconnect periodically until a 
connection is re-established with the broker.  We've noticed however, that 
sometimes these reconnection attempts hang *indefinitely/forever* despite a 
configured RequestedConnectionTimeout value of 30 seconds.

After months of seeing this problem every few weeks because of various 
network hiccups, we were finally able to capture a snapshot and then 
perform an inspection of the memory dump using WinDbg.  Here's the stack 
trace (which uses the production RabbitMQ .NET client v3.2.1):

000000001adde530 000007fef77ad01a System.Net.Sockets.Socket.Receive(Byte[], 
Int32, Int32, System.Net.Sockets.SocketFlags, 
System.Net.Sockets.SocketError ByRef)
000000001adde5d0 000007fef77ace88 
System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32)
000000001adde650 000007fef777294c 
System.Net.FixedSizeReader.ReadPacket(Byte[], Int32, Int32)
000000001adde6a0 000007fef77d3803 
System.Net.Security.SslState.StartReceiveBlob(Byte[], 
System.Net.AsyncProtocolRequest)
000000001adde6e0 000007fef77d1f09 
System.Net.Security.SslState.StartSendBlob(Byte[], Int32, 
System.Net.AsyncProtocolRequest)
000000001adde760 000007fef77d3d26 
System.Net.Security.SslState.ProcessReceivedBlob(Byte[], Int32, 
System.Net.AsyncProtocolRequest)
000000001adde7d0 000007fef77d3816 
System.Net.Security.SslState.StartReceiveBlob(Byte[], 
System.Net.AsyncProtocolRequest)
000000001adde810 000007fef77d1f09 
System.Net.Security.SslState.StartSendBlob(Byte[], Int32, 
System.Net.AsyncProtocolRequest)
000000001adde890 000007fef77d1b85 
System.Net.Security.SslState.ForceAuthentication(Boolean, Byte[], 
System.Net.AsyncProtocolRequest)
000000001adde900 000007fef77d1856 
System.Net.Security.SslState.ProcessAuthentication(System.Net.LazyAsyncResult)

*RabbitMQ.Client.Impl.SocketFrameHandler_0_9..ctor(RabbitMQ.Client.AmqpTcpEndpoint, 
ObtainSocket, Int32)*
000000001addea00 000007fe9a28c5e1 
RabbitMQ.Client.Framing.Impl.v0_9_1.ProtocolBase.CreateFrameHandler(RabbitMQ.Client.AmqpTcpEndpoint, 
ObtainSocket, Int32)
000000001addea50 000007fe9a28c245 
RabbitMQ.Client.ConnectionFactory.FollowRedirectChain(Int32, 
System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,Int32>, 
System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,System.Exception>, 
RabbitMQ.Client.AmqpTcpEndpoint[] ByRef, RabbitMQ.Client.AmqpTcpEndpoint)
000000001addeb50 000007fe9a28c05f 
RabbitMQ.Client.ConnectionFactory.CreateConnection(Int32, 
System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,Int32>, 
System.Collections.Generic.IDictionary`2<RabbitMQ.Client.AmqpTcpEndpoint,System.Exception>, 
RabbitMQ.Client.AmqpTcpEndpoint[])
000000001addebe0 000007fe9a28bf51 
RabbitMQ.Client.ConnectionFactory.CreateConnection(Int32)
000000001addec70 000007fe9a28bd6f *** ERROR: Module load completed but 
symbols could not be loaded for Accounting.UsageHost.dll

If you follow the stack trace through the .NET client source code, you'll 
notice that a few of the methods are missing.  These have been optimized 
away by the compiler when using a "Release" build.

I have highlighted exactly where the problem is above and if you follow the 
source code, you'll find the problem right here:
*http://hg.rabbitmq.com/rabbitmq-dotnet-client/file/e399c21792d7/projects/client/RabbitMQ.Client/src/client/api/SslHelper.cs#l97*

The problem is that there is *no timeout* value configured for SSL 
connections.  This means that the SSL connection attempt will hang 
indefinitely.  And since the connection doesn't exist yet, there's no way 
for any kind of heartbeat thread to tear things down.

The .NET docs also state that the default read/write timeout is set to 
Infinite by default<http://msdn.microsoft.com/en-us/library/system.net.security.sslstream.readtimeout(v=vs.110).aspx>
.

The simple fix is as follows:
sslStream.ReadTimeout = connectionTimeout;
sslStream.WriteTimeout=connectionTimout;




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140216/ba4eca34/attachment.html>