[rabbitmq-discuss] Consumers stops to pick up messages after long inactivity

[Emile Joubert]

Hi Neha,

It is possible that the combination of broker restarts and automatic 
connection re-establishment is part of the problem. Are you able to 
repeat the test over a weekend without restarting the broker and see 
whether the outcome is any different?

Regards

Emile

On 18/01/11 18:01, Neha wrote:
 > Hi Emile
 >
 > I tracked the status of the connection by logging the value of
 > 'connection.isOpen()'. I observed that somewhere in between it became
 > false, i.e. the connection was closed. It doesn't give me any
 > exceptions or the reason why the connection was closed. The connection
 > is not null, thus it still exits but is closed. The broker log file
 > shows that after the connection is closed it retries to establish the
 > shared rabbit mq connection and is successful. It restarts the broker
 > too. I have a logic in place where after the broker is restarted it
 > puts the messages back in the queue. Once the connection is
 > reestablished and broker is restarted the 'connection.isOpen()' always
 > remains false. The broker still publishes messages on the queue but
 > the consumer is unable to consume it.
 >
 > I restarted the consumer and it starts picking up the messages on the
 > queue.
 >
 > Is there a way to find out the reason why the connection was closed?
 > Also once the connection is re-established again, is it necessary to
 > restart the consumer as well?


On 13/01/11 19:46, Neha wrote:
> Hi Emile
>
> Thanks for the reply.
> I checked the broker logfiles and it has a few
> "AutoFailoverConnectionFactory" messages. But it again connects
> successfully.
> I checked the timeout and that is listed as 0.
> To check the clients connected to the broker I'll have to wait for
> Monday and see if the brokers considers the consumers to be connected.
>
> Will keep this post updated with the cause of the problem.
>
> Thanks
> Neha
>
> On Jan 11, 10:47 am, Emile Joubert<em... at rabbitmq.com>  wrote:
>> Hi Neha,
>>
>> On 10/01/11 21:57, Neha wrote:
>>
>>
>>
>>> I have an application which uses rabbitmq. It has messages being sent
>>> from Monday to Friday and on the weekend there are no messages sent.
>>
>>> The application works fine on regular days, i.e producers puts the
>>> message on the queue and the consumer receives the message. But every
>>> Monday, after the weekend the consumer starts behaving strangely. It
>>> stops receiving messages even though the queue has messages present.
>>> The heartbeat of the consumer all this time is working fine but it
>>> still doesn't receive all these messages on the queue. To overcome
>>> this problem I have to restart the consumers every Monday, and as soon
>>> as I do that, all the messages that are present in the queue starts
>>> getting consumed.
>>
>>> Looking at a problem, I feel it might be due to inactivity during the
>>> weekend. I wanted to know if anyone else has faced this problem
>>> wherein the consumer stops consuming after a long duration of
>>> inactivity? Is there a setting to be tweaked to increase this period?
>>
>> Long periods of inactivity should have no effect. The problem is almost
>> certainly network-related. Here are some things to check.
>>
>> Is there anything of interest in the broker logfile?
>>
>> As there any device along the network path that might have caused the
>> connections to be reset?
>>
>> Does the broker consider the clients to be connected on Monday mornings?
>> "rabbitmqctl list_connections" and "rabbitmqctl list_consumers" can help
>> determine this. The logfile will show when any connections were
>> terminated. Also verify the connection timeout (rabbitmqctl
>> list_connections timeout).
>>
>> You may want to try using a different connection heartbeat value and
>> check whether the message consumers are able to recover from network
>> interruptions automatically.