<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hi Michael,<div><br></div><div>Thanks very much for your response. I have a bit more data about this that could be interesting.</div><div><br></div><div>Our producer node dysfunction seems to be correlated with a heartbeat timeout event:</div><div><br></div><div><div>5/7/14 12:44:11.000 AM<span class="Apple-tab-span" style="white-space:pre"> </span></div><div>=INFO REPORT==== 7-May-2014::00:44:11 ===</div><div>accepting AMQP connection <0.17458.855> (172.20.153.34:36684 -> 172.20.153.185:5672)</div><div><br></div><div><br></div><div>5/7/14 12:44:11.000 AM<span class="Apple-tab-span" style="white-space:pre"> </span></div><div>=ERROR REPORT==== 7-May-2014::00:44:11 ===</div><div>closing AMQP connection <0.30506.852> (172.20.153.34:37769 -> 172.20.153.185:5672):</div><div>{heartbeat_timeout,running}</div><div><br></div><div><br></div><div>If we check netstat on the producer / rabbit-server sides, we see a very large Send-Q value on the producer, and nothing on the rabbit-server side; the producer Send-Q size just continues to grow larger, with no response from the server:</div><div><br></div><div>producer:</div><div><div>[ccpmer-po-cb48-p ~ ]$ netstat -an | grep 5672</div><div>tcp 0 1066985 ::ffff:172.20.153.34:36684 ::ffff:172.20.153.185:5672 ESTABLISHED </div><div>tcp 1 0 ::ffff:172.20.153.34:60075 ::ffff:172.20.149.41:15672 CLOSE_WAIT </div><div><br></div><div>rabbit side:</div><div>[ccpmer-po-cb176-p ~ ]$ netstat -an | grep 5672 | grep 34</div><div>tcp 0 0 ::ffff:172.20.153.185:5672 ::ffff:172.20.153.34:36684 ESTABLISHED </div><div><br></div><div>producer Send-Q keeps growing.</div><div>[ccpmer-po-cb48-p ~ ]$ netstat -an | grep 5672</div><div>tcp 0 1074230 ::ffff:172.20.153.34:36684 ::ffff:172.20.153.185:5672 ESTABLISHED </div><div>tcp 1 0 ::ffff:172.20.153.34:60075 ::ffff:172.20.149.41:15672 CLOSE_WAIT</div></div><div><br></div><div><br></div><div>Again we’re running the 3.2.2 version of the server - I do see that bugfix 25457 fixed a similar issue in the 3.0.3 release, but we’re past that now.</div><div><br></div><div>Is this still indicative of the socketWrite timeout issue you referred me to in your last response, or could this be something else? Is there more data you’d like to see that could help diagnose this further?</div><div><br></div><div>Are there any plans to fix <a href="https://github.com/rabbitmq/rabbitmq-java-client/issues/11">https://github.com/rabbitmq/rabbitmq-java-client/issues/11</a> soon? There’s not another more recent version of rabbit we could upgrade to that would help us, is there? </div><div><br></div><div>Thanks again,</div><div><br></div><div>Jeff</div><div><br></div><div><br></div><div><br></div><div><div>On Apr 28, 2014, at 6:31 PM, Michael Klishin <<a href="mailto:mklishin@gopivotal.com">mklishin@gopivotal.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">On 29 April 2014 at 02:29:30, Jeff Doto (<a href="mailto:jeffdoto@gmail.com">jeffdoto@gmail.com</a>) wrote:<br><blockquote type="cite"><blockquote type="cite">We’re wondering if anyone has encountered an issue similar <br></blockquote>to this running the versions of Rabbit server / client we are. <br>Are there any suggestions for what we can do to prevent this? Could <br>upgrading the server version help?<br></blockquote><br>This is a known Java client (or, well, JDK Socket) limitation:<br><a href="http://markmail.org/thread/w7ay4hslxwqirfsc">http://markmail.org/thread/w7ay4hslxwqirfsc</a> <br>-- <br>MK <br><br>Software Engineer, Pivotal/RabbitMQ<br></blockquote></div><br></div></body></html>