[rabbitmq-discuss] RabbitMQ blocking issue

[Carroll James (Nokia-LC/Malvern)]

As far as where it's hanging on the client side: I don't have the exact rabbit call stack (I'll save off the jstack next time) but it's inside the amqp client code and inside of the native socketWrite call. Both the send buffer on the client side and the recv buffer on the server side are full. While the server is still maintaining the connection it's not reading from the socket. Since the client is written in terms of blocking java.io, the client hangs waiting for the server to read the data.

-----Original Message-----
From: Konar Peter (Nokia-LC/Chicago)
Sent: Friday, February 08, 2013 12:56 PM
To: ext Matthias Radestock; Discussions about RabbitMQ; Carroll James (Nokia-LC/Malvern); Mccuiston Mitchell (Nokia-LC/Chicago); Lieber Dick (Nokia-LC/Chicago)
Cc: Johnson Anthony (Nokia-LC/Alpharetta); Yu Jerry.3 (Nokia-LC/Alpharetta)
Subject: RE: [rabbitmq-discuss] RabbitMQ blocking issue

Matthias,
        Adding more people to the discussion.

- Where exactly does the client hang? Does it get stuck in a rabbitmq client API call? If so, which one?
I will get you a stack trace as soon as I can capture a failed case.

-It is possible that the TCP connections have been disrupted.
Could be we are living in the virtual world, so I am not sure what the connectivity does? (Jerry/Anthony - can you answer?)

Are AMQP heartbeats enabled on the connections? If not then please do that. Or upgrade the server 3.0.x which enables heartbeats automatically.
No, we are upgrading to 3.0.2 to hopefully get us this.

Thanks
Peter Konar

-----Original Message-----
From: ext Matthias Radestock [mailto:matthias at rabbitmq.com]
Sent: Thursday, February 07, 2013 5:41 PM
To: Discussions about RabbitMQ
Cc: Konar Peter (Nokia-LC/Chicago)
Subject: Re: [rabbitmq-discuss] RabbitMQ blocking issue

Peter,

On 07/02/13 23:00, Konar Peter (Nokia-LC/Chicago) wrote:
> We have an issue with our RabbitMQ clients not publishing to our
> RabbitMQ server. We see that the client hangs approx. every 2 hours
> and never recovers.
>
> Versions: Client: amqp-client-2.8.7.jar  ,
> spring-amqp-1.1.1.RELEASE.jar server:
> noarch/rabbitmq-server-2.8.7-1.noarch.rpm

Where exactly does the client hang? Does it get stuck in a rabbitmq client API call? If so, which one?

> On the client side we see that our application stops working. When we
> query the port we see: [pkonar at ajtrafp067 ~]$ netstat -an | grep
> 5672 tcp        0 194017 ::ffff:10.196.42.93:47657
> ::ffff:10.196.42.43:5672    FIN_WAIT1 tcp        0   2722
> ::ffff:10.196.42.93:46361   ::ffff:10.196.42.43:5672    ESTABLISHED
>
> The FIN_WAIT1 is from the application restart and never recovers the
> port. Meaning never releases the port from FIN_WAIT1.

IIRC this is normal tcp behaviour. I'm pretty sure the O/S will release the port eventually. Could take a while though, possibly hours.

> After restarting the application we see a new connection created the
> works, but we would prefer not restarting the app every 2 hours.

It is possible that the TCP connections have been disrupted.

Are AMQP heartbeats enabled on the connections? If not then please do that. Or upgrade the server 3.0.x which enables heartbeats automatically.

Heartbeats ensure that disrupted tcp connections get detected promptly.
They also help keeping connections alive in the presence of firewalls which may close them when idle for a while (thus being one source of disrupted connections).

> Server side [pkonar at ajtrafp017 ~]$ sudo rabbitmqctl list_connections
> name last_blocked_age last_blocked_by Listing connections ...
> 10.196.42.90:35923 -> 10.196.42.43:5672 497412.591327   flow
> [etc]

That all looks perfectly normal.

Matthias.

The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.