[rabbitmq-discuss] RabbitMQ blocking issue

[Konar Peter (Nokia-LC/Chicago)]

Matthias,
        Adding more people to the discussion.

- Where exactly does the client hang? Does it get stuck in a rabbitmq client API call? If so, which one?
I will get you a stack trace as soon as I can capture a failed case.

-It is possible that the TCP connections have been disrupted.
Could be we are living in the virtual world, so I am not sure what the connectivity does? (Jerry/Anthony - can you answer?)

Are AMQP heartbeats enabled on the connections? If not then please do that. Or upgrade the server 3.0.x which enables heartbeats automatically.
No, we are upgrading to 3.0.2 to hopefully get us this.

Thanks
Peter Konar

-----Original Message-----
From: ext Matthias Radestock [mailto:matthias at rabbitmq.com]
Sent: Thursday, February 07, 2013 5:41 PM
To: Discussions about RabbitMQ
Cc: Konar Peter (Nokia-LC/Chicago)
Subject: Re: [rabbitmq-discuss] RabbitMQ blocking issue

Peter,

On 07/02/13 23:00, Konar Peter (Nokia-LC/Chicago) wrote:
> We have an issue with our RabbitMQ clients not publishing to our
> RabbitMQ server. We see that the client hangs approx. every 2 hours
> and never recovers.
>
> Versions: Client: amqp-client-2.8.7.jar  ,
> spring-amqp-1.1.1.RELEASE.jar server:
> noarch/rabbitmq-server-2.8.7-1.noarch.rpm

Where exactly does the client hang? Does it get stuck in a rabbitmq client API call? If so, which one?

> On the client side we see that our application stops working. When we
> query the port we see: [pkonar at ajtrafp067 ~]$ netstat -an | grep
> 5672 tcp        0 194017 ::ffff:10.196.42.93:47657
> ::ffff:10.196.42.43:5672    FIN_WAIT1 tcp        0   2722
> ::ffff:10.196.42.93:46361   ::ffff:10.196.42.43:5672    ESTABLISHED
>
> The FIN_WAIT1 is from the application restart and never recovers the
> port. Meaning never releases the port from FIN_WAIT1.

IIRC this is normal tcp behaviour. I'm pretty sure the O/S will release the port eventually. Could take a while though, possibly hours.

> After restarting the application we see a new connection created the
> works, but we would prefer not restarting the app every 2 hours.

It is possible that the TCP connections have been disrupted.

Are AMQP heartbeats enabled on the connections? If not then please do that. Or upgrade the server 3.0.x which enables heartbeats automatically.

Heartbeats ensure that disrupted tcp connections get detected promptly.
They also help keeping connections alive in the presence of firewalls which may close them when idle for a while (thus being one source of disrupted connections).

> Server side [pkonar at ajtrafp017 ~]$ sudo rabbitmqctl list_connections
> name last_blocked_age last_blocked_by Listing connections ...
> 10.196.42.90:35923 -> 10.196.42.43:5672 497412.591327   flow
> [etc]

That all looks perfectly normal.

Matthias.

The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.