[rabbitmq-discuss] EndOfStreamException after 15 minutes of inactivity

Thu Jun 7 22:27:03 BST 2012

Hey Tim,

Enabling heartbeats seems to have helped with my issue.  Making this change triggered the Socket ObjectDisposedException problem discussed at http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-March/018767.html.  Updating to the most recent build on NuGet appears to have cleared that issue up as well.  Will monitor for a few days before declaring victory.

Thank you for your help,

Walter Moore
Head of Transactional Data Exchange
INTL FCStone Inc
230 S. LaSalle Street
Suite 10-500
Chicago IL  60604
Direct 312-789-2538

-----Original Message-----
From: Tim Watson [mailto:watson.timothy at gmail.com] 
Sent: Thursday, June 07, 2012 5:46 AM
To: Walter Moore
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] EndOfStreamException after 15 minutes of inactivity

On 7 Jun 2012, at 10:08, Tim Watson wrote:

> On 06/06/12 22:05, Tim Watson wrote:
>> On 06/06/2012 16:18, Walter Moore wrote:
>>> Hi Tim,
>>> 
>>> I am not explicitly setting the TTL when creating the queue. Looking 
>>> over the TTL docs, it looks like declaring the queue with "x-expires"
>>> set to a very large (long-long-int) number would effectively fix the 
>>> issue. I'll give that a shot this morning.
>>> 
>> 
>> The x-expires extension controls queue lifetime, whereas x-message-ttl controls the dwell time for messages before they get discarded. I'm not convinced this has anything to do with the behaviour you specified, as if it isn't currently set then nothing should be causing the queue to be removed (and as you've not said otherwise, I'm assuming that the queue survives this disconnection of the consumer).
>> 
>>> I have confirmed via logging that Heartbeat = 0 in the .Net client 
>>> connection.
>>> 
>> 
>> Ok - that seemed like a logical candidate but perhaps it's a red herring. It does 'sound' like some kind of timeout/expiry is happening, but if there's no heartbeat set then perhaps that's not the case.
>> 
> 
> Hi Walter,
> 
> Can you please try *enabling* heartbeats - as their whole purpose is to ensure that the connection stays alive for long enough - and let us know if that resolves the problem?
> 

Actually I've mis-stated that. AMQP heartbeats are also used by the broker to deal with 'misbehaving' clients, but either way, they should help with your situation I hope.

> Cheers,
> Tim
> 
>>> It is probably also worth mentioning that this is occurring on an 
>>> SSL connection. The exchange is manually configured via the 
>>> Management plugin web interface using the following parameters:
>>> 
>>> Type: topic
>>> 
>>> Durable: true
>>> 
>>> The queue is created with the following parameters:
>>> 
>>> QueueName: <ApplicationName>.<ProcessName>.<ConsumerName>
>>> 
>>> Durable: true
>>> 
>>> Exclusive: false
>>> 
>>> AutoDelete: false
>>> 
>>> Arguments: null
>>> 
>>> The queue is bound to a number of filters of the form 
>>> <ApplicationName>.<ProcessName>.<ProducerName> which can be on 
>>> several different exchanges. The queue name, routing keys, and key 
>>> bindings are all managed with an internally developed toolset.
>>> 
>> 
>> I'd be surprised to find that any of those (latter) things is at the root of this behaviour.
>> 
>>> Thank you,
>>> 
>>> *Description: Description: Description: INTLFCStone*
>>> 
>>> **
>>> 
>>> *Walter Moore*
>>> 
>>> Head of Transactional Data Exchange
>>> 
>>> INTL FCStone Inc
>>> 
>>> 230 S. LaSalle Street
>>> 
>>> Suite 10-500
>>> 
>>> Chicago IL 60604
>>> 
>>> *Direct 312-789-2538*
>>> 
>>> *From:*Tim Watson [mailto:tim at rabbitmq.com]
>>> *Sent:* Wednesday, June 06, 2012 2:32 AM
>>> *To:* Walter Moore
>>> *Cc:* rabbitmq-discuss at lists.rabbitmq.com
>>> *Subject:* Re: [rabbitmq-discuss] EndOfStreamException after 15 
>>> minutes of inactivity
>>> 
>>> On 06/06/12 07:59, Tim Watson wrote:
>>> 
>>> On 05/06/12 21:24, Walter Moore wrote:
>>> 
>>> Hello,
>>> 
>>> I have a long running, durable, non-auto-delete, non-exclusive, 
>>> RabbitMq consumer (.Net client) that works perfectly unless a 15 
>>> minute (+- approximately 3 seconds) period of inactivity occurs. 
>>> After 15 minutes of messaging inactivity a 
>>> "System.IO.EndOfStreamException: SharedQueue closed" exception is 
>>> thrown (and continues to be thrown for each subsequent call to 
>>> <model>.BasicConsume()). Is this the expected behavior? I am using NoAck = true for each dequeue..
>>> 
>>> Hi Walter,
>>> 
>>> When you say 'non-auto-delete' do you mean to say there is not TTL 
>>> specified on the messages?
>>> I also wondered if you have AMQP heartbeats enabled - see 
>>> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-May/020236
>>> .html for a discussion of this - as that could cause a connection to 
>>> be terminated after a specified period of inactivity.
>>> 
>>> Looking further at the client code, I can see that the heartbeat is 
>>> implemented on the client as well as the server. If this times out, 
>>> we do indeed get an EndOfStreamException (with a message like 
>>> "Heartbeat missing with heartbeat == <n> seconds") - see
>>> http://hg.rabbitmq.com/rabbitmq-dotnet-client/file/default/projects/
>>> client/RabbitMQ.Client/src/client/impl/ConnectionBase.cs#l541
>>> for the details.
>>> 
>>> Looking at the SharedQueue class, I can see that once closed, it 
>>> will continuously throw this exception whenever you try to dequeue. 
>>> So this looks like a possible cause to me. Can you verify whether or 
>>> not you've got AMQP heartbeat timeouts enabled?
>>> 
>>> Cheers,
>>> Tim
>>> 
>>> 
>>> I'll have a look in the .NET client code, as this doesn't seem like 
>>> the exception class I'd expect to be thrown in a heartbeat timeout 
>>> situation, but I thought it was worth mentioning just in case.
>>> 
>>> If possible, can you tell us exactly (scratching any private data 
>>> such as the broker, queue and/or exchange names of course) what 
>>> parameters you're passing to declare the queue and exchange(s) as 
>>> well as the consumer set up?
>>> 
>>> Cheers,
>>> Tim
>>> 
>>> Thank you,
>>> 
>>> *Description: Description: Description: INTLFCStone*
>>> 
>>> **
>>> 
>>> *Walter Moore*
>>> 
>>> Head of Transactional Data Exchange
>>> 
>>> INTL FCStone Inc
>>> 
>>> 230 S. LaSalle Street
>>> 
>>> Suite 10-500
>>> 
>>> Chicago IL 60604
>>> 
>>> *Direct 312-789-2538*
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> 
>>> rabbitmq-discuss mailing list
>>> 
>>> rabbitmq-discuss at lists.rabbitmq.com 
>>> <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>>> 
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> 
>>> rabbitmq-discuss mailing list
>>> 
>>> rabbitmq-discuss at lists.rabbitmq.com 
>>> <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>>> 
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> 
>> 
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss