[rabbitmq-discuss] RabbitMQ blocking issue

[Matthias Radestock]

Peter,

On 07/02/13 23:00, Konar Peter (Nokia-LC/Chicago) wrote:
> We have an issue with our RabbitMQ clients not publishing to our
> RabbitMQ server. We see that the client hangs approx. every 2 hours
> and never recovers.
>
> Versions: Client: amqp-client-2.8.7.jar  ,
> spring-amqp-1.1.1.RELEASE.jar server:
> noarch/rabbitmq-server-2.8.7-1.noarch.rpm

Where exactly does the client hang? Does it get stuck in a rabbitmq 
client API call? If so, which one?

> On the client side we see that our application stops working. When we
> query the port we see: [pkonar at ajtrafp067 ~]$ netstat -an | grep
> 5672 tcp        0 194017 ::ffff:10.196.42.93:47657
> ::ffff:10.196.42.43:5672    FIN_WAIT1 tcp        0   2722
> ::ffff:10.196.42.93:46361   ::ffff:10.196.42.43:5672    ESTABLISHED
>
> The FIN_WAIT1 is from the application restart and never recovers the
> port. Meaning never releases the port from FIN_WAIT1.

IIRC this is normal tcp behaviour. I'm pretty sure the O/S will release 
the port eventually. Could take a while though, possibly hours.

> After restarting the application we see a new connection created the works,
> but we would prefer not restarting the app every 2 hours.

It is possible that the TCP connections have been disrupted.

Are AMQP heartbeats enabled on the connections? If not then please do 
that. Or upgrade the server 3.0.x which enables heartbeats automatically.

Heartbeats ensure that disrupted tcp connections get detected promptly. 
They also help keeping connections alive in the presence of firewalls 
which may close them when idle for a while (thus being one source of 
disrupted connections).

> Server side [pkonar at ajtrafp017 ~]$ sudo rabbitmqctl list_connections
> name last_blocked_age last_blocked_by Listing connections ...
> 10.196.42.90:35923 -> 10.196.42.43:5672 497412.591327   flow
> [etc]

That all looks perfectly normal.

Matthias.