[rabbitmq-discuss] Writes to Rabbit Occasionally Hang Indefinitely

Wed May 7 17:41:21 BST 2014

Hi Michael,

Thanks very much for your response.  I have a bit more data about this that could be interesting.

Our producer node dysfunction seems to be correlated with a heartbeat timeout event:

5/7/14 12:44:11.000 AM	
=INFO REPORT==== 7-May-2014::00:44:11 ===
accepting AMQP connection <0.17458.855> (172.20.153.34:36684 -> 172.20.153.185:5672)

5/7/14 12:44:11.000 AM	
=ERROR REPORT==== 7-May-2014::00:44:11 ===
closing AMQP connection <0.30506.852> (172.20.153.34:37769 -> 172.20.153.185:5672):
{heartbeat_timeout,running}

If we check netstat on the producer / rabbit-server sides, we see a very large Send-Q value on the producer, and nothing on the rabbit-server side; the producer Send-Q size just continues to grow larger, with no response from the server:

producer:
[ccpmer-po-cb48-p ~ ]$ netstat -an | grep 5672
tcp        0 1066985 ::ffff:172.20.153.34:36684  ::ffff:172.20.153.185:5672  ESTABLISHED 
tcp        1      0 ::ffff:172.20.153.34:60075  ::ffff:172.20.149.41:15672  CLOSE_WAIT  

rabbit side:
[ccpmer-po-cb176-p ~ ]$ netstat -an | grep 5672 | grep 34
tcp        0      0 ::ffff:172.20.153.185:5672  ::ffff:172.20.153.34:36684  ESTABLISHED 

producer Send-Q keeps growing.
[ccpmer-po-cb48-p ~ ]$ netstat -an | grep 5672
tcp        0 1074230 ::ffff:172.20.153.34:36684  ::ffff:172.20.153.185:5672  ESTABLISHED 
tcp        1      0 ::ffff:172.20.153.34:60075  ::ffff:172.20.149.41:15672  CLOSE_WAIT

Again we’re running the 3.2.2 version of the server - I do see that bugfix 25457 fixed a similar issue in the 3.0.3 release, but we’re past that now.

Is this still indicative of the socketWrite timeout issue you referred me to in your last response, or could this be something else?  Is there more data you’d like to see that could help diagnose this further?

Are there any plans to fix https://github.com/rabbitmq/rabbitmq-java-client/issues/11 soon?  There’s not another more recent version of rabbit we could upgrade to that would help us, is there?  

Thanks again,

Jeff

On Apr 28, 2014, at 6:31 PM, Michael Klishin <mklishin at gopivotal.com> wrote:

> On 29 April 2014 at 02:29:30, Jeff Doto (jeffdoto at gmail.com) wrote:
>>> We’re wondering if anyone has encountered an issue similar  
>> to this running the versions of Rabbit server / client we are.  
>> Are there any suggestions for what we can do to prevent this? Could  
>> upgrading the server version help?
> 
> This is a known Java client (or, well, JDK Socket) limitation:
> http://markmail.org/thread/w7ay4hslxwqirfsc 
> --  
> MK  
> 
> Software Engineer, Pivotal/RabbitMQ

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140507/fbfb2fa9/attachment.html>