[rabbitmq-discuss] Shutdown Listener on Java Client API

[Dave Greggory]

>> We had something similar happen again last night. For some reason, one of our producers lost the connection to RabbitMQ at 7:47 PM
> 
> How did you observe this?

The client web app started dropping messages at 7:47 PM. The app queues messages in internal buffer and a separate thread goes and empties this queue by sending them to rabbitMQ. This async thread prevents any blocking message sends from affecting the rest of functionality. Since we observed dropped messages we can presume that the internal buffer / queue was full and the message send thread either blocked or slowed to a crawl. No errors or exceptions in that thread. No CPU or memory or disk use spikes. We kept dropping messages from 7:47 till 8:58 PM when it detected the connection loss and automatically re-connected. 

Can't say for sure that the connection was lost at 7:47 PM but all things point to it. 

I think I mentioned previously that RabbitMQ nodes stayed up. No message build (our queues are mostly empty all the time as our consumer is really fast). No errors in RabbitMQ logs. 

>> but the shutDownCompleted event did not fire on that producer till 8:58 PM.
> 
> It's possible for a stale network connection to go undetected for a long time. The presence of firewalls and loadbalancers in the path seems to increase the likelihood of this happening.

True, the problem is probably not in RabbitMQ... But we'd like to detect it as quickly as possible when it happens to take preventive action. 

>> Any thoughts? Is there something in the java client api that could prevent the shutdownEvent from firing quickly?
> 
> No. The most likely cause is that the tcp stack simply took that long to  notice the connection was dead.
> 
>> AMQP Heartbeats were mentioned in another thread, would that help and
>> how do we enable these?
> 
> Yes, heartbeats would help. Enable them with http://www.rabbitmq.com/releases/rabbitmq-java-client/v2.1.0/rabbitmq-java-client-javadoc-2.1.0/com/rabbitmq/client/ConnectionFactory.html#setRequestedHeartbeat(int)
> 
> You could also try playing with TCP's 

Thanks will try heart beats, TCP keep alive's are already enabled by the load balancer.