<div dir="ltr"><div><div>Thanks for the answer Michael,<br><br></div><div>Configuring SO_LINGER didn't help for socketWrite timeout. Also it seems it affects only "close" operation and also there is a code in SocketFrameHandler which already sets SO_LINGER for socket on close (1 second default):<br>
<br> public void close() {<br> try { _socket.setSoLinger(true, SOCKET_CLOSING_TIMEOUT); } catch (Exception _) {}<br> try { flush(); } catch (Exception _) {}<br> try { _socket.close(); } catch (Exception _) {}<br>
}<br><br></div><div>But I even cannot close connection by myself in a separate thread - it's blocked in SocketFrameHandler#writeFrame method:<br></div><div><br>Thread [Thread-0] (Suspended) <br> owns: CommandAssembler (id=42) <br>
owns: Object (id=41) <br> waiting for: DataOutputStream (id=34) <br> SocketFrameHandler.writeFrame(Frame) line: 137 <br> AMQConnection.writeFrame(Frame) line: 480 <br> AMQCommand.transmit(AMQChannel) line: 102 <br>
AMQConnection$1(AMQChannel).quiescingTransmit(AMQCommand) line: 316 <br> AMQConnection$1(AMQChannel).quiescingTransmit(Method) line: 298 <br> AMQConnection$1(AMQChannel).quiescingRpc(Method, AMQChannel$RpcContinuation) line: 233 <br>
AMQConnection.close(int, String, boolean, Throwable, int, boolean) line: 800 <br> AMQConnection.close(int, String, int) line: 724 <br> AMQConnection.close(int) line: 710 <br> Send$2.run() line: 118 <br>
Thread.run() line: 724 <br><br></div><div>I googled a bit again, here someone asks sun to provide socket write timeout (1997 year) : <br><a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4031100" target="_top" rel="nofollow" link="external">http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4031100</a><br>
</div><div>but the answer is to use nio SocketChannels or workaround with different thread for read and write.<br><br></div><div>Here someone asks ActiveMQ to provide such write timeout:<br><a href="https://issues.apache.org/jira/browse/AMQ-1993" target="_top" rel="nofollow" link="external">https://issues.apache.org/jira/browse/AMQ-1993</a><br>
</div><div>and the answer is to use "transport.soWriteTimeout" parameter which is managed by ActiveMQ itself<br><a href="http://activemq.apache.org/maven/apidocs/org/apache/activemq/transport/WriteTimeoutFilter.html" target="_top" rel="nofollow" link="external">http://activemq.apache.org/maven/apidocs/org/apache/activemq/transport/WriteTimeoutFilter.html</a><br>
<br></div><div>As for maximum packet age - the client connection has state "ESTABLISHED" during all 15 minutes, not TIME_WAIT.<br></div><div><br>There is a setting in linux which can configure TCP retransmissions timeouts : /proc/sys/net/ipv4/tcp_retries2. By default the value is 15 and it's about 900 seconds:<br>
<a href="http://stackoverflow.com/questions/5907527/application-control-of-tcp-retransmission-on-linux" target="_top" rel="nofollow" link="external">http://stackoverflow.com/questions/5907527/application-control-of-tcp-retransmission-on-linux</a><br></div><div>It's not recommended to set the value < 100 seconds and we cannot actually change it because there are lots of other applications on the machine and we do not know consequences of such changes.<br>
<br></div><div>So for me it looks like the RabbitMQ client library should provide possibility to set such write timeout - maybe using the same Heartbeat thread (for now this thread is blocked on "writeFrame" method like all other threads waiting freeing the lock on outputStream object).<br>
<br></div><div>By the way I still cannot imagine any workaround for this issue but it's really critical for us. Do you think there are some?<br></div><div><br></div>Regards,<br></div>Oleg<br><div class="gmail_extra"><br>
<br><div class="gmail_quote">2013/10/7 Michael Klishin [via RabbitMQ] <span dir="ltr"><<a href="/user/SendEmail.jtp?type=node&node=30320&i=0" target="_top" rel="nofollow" link="external">[hidden email]</a>></span><br>
<blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class="h5">
On oct 7, 2013, at 4:26 p.m., Oleg Lyalikov <<a href="http://user/SendEmail.jtp?type=node&node=30283&i=0" rel="nofollow" link="external" target="_blank">[hidden email]</a>> wrote:
<br><div><div><div class='shrinkable-quote'><br>> "main" prio=10 tid=0xf7505800 nid=0x2c3e runnable [0xf7728000]
<br>> java.lang.Thread.State: RUNNABLE
<br>> at java.net.SocketOutputStream.socketWrite0(Native Method)
<br>> at
<br>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
<br>> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
<br>> at
<br>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
<br>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
<br>> - locked <0xb4532608> (a java.io.BufferedOutputStream)
<br>> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
<br>> at
<br>> com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.java:142)
<br>> at
<br>> com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:488)
<br>> at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:125)
<br>> at
<br>> com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:316)
<br>> - locked <0xb4532ee8> (a java.lang.Object)
<br>> at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:292)
<br>> - locked <0xb4532ee8> (a java.lang.Object)
<br>> at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:634)
<br>> at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:617)
<br>> at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:608)
</div></div>If SocketOutputStream#socketWrite0 takes minutes to timeout, it may be
</div>worth trying setting SO_LINGER on the socket (SO_TIMEOUT sounds like
<br>what you want but I believe it only covers read operations):
<br><br><a href="http://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSoLinger(boolean,%20int" rel="nofollow" link="external" target="_blank">http://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSoLinger(boolean,%20int</a>)
<br><br>To do so, for example, you can subclass default ConnectionFactory and override
<br>#configureSocket:
<br><br><a href="http://hg.rabbitmq.com/rabbitmq-java-client/file/46578678645e/src/com/rabbitmq/client/ConnectionFactory.java#l476" rel="nofollow" link="external" target="_blank">http://hg.rabbitmq.com/rabbitmq-java-client/file/46578678645e/src/com/rabbitmq/client/ConnectionFactory.java#l476</a><br>
<br>Note that SO_LINGER is not free of downsides:
<br><br><a href="http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable" rel="nofollow" link="external" target="_blank">http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable</a><br>
<a href="http://stackoverflow.com/questions/3757289/tcp-option-so-linger-zero-when-its-required" rel="nofollow" link="external" target="_blank">http://stackoverflow.com/questions/3757289/tcp-option-so-linger-zero-when-its-required</a><br>
<br>You can try reducing maximum packet age for the server, this should reduce the amount of
<br>time spent in TIME_WAIT.
<br><br>MK
<br><br><br><br><br></div></div>_______________________________________________
<br>rabbitmq-discuss mailing list
<br><a href="http://user/SendEmail.jtp?type=node&node=30283&i=1" rel="nofollow" link="external" target="_blank">[hidden email]</a>
<br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" rel="nofollow" link="external" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br><div><br><img src="http://images/icon_attachment.gif"> <b>signature.asc</b> (506 bytes) <a href="http://rabbitmq.1065348.n5.nabble.com/attachment/30283/0/signature.asc" rel="nofollow" link="external" target="_blank">Download Attachment</a></div>
<br>
<br>
<hr color="#cccccc" noshade size="1">
<div style="color:rgb(68,68,68);font:12px tahoma,geneva,helvetica,arial,sans-serif">
<div style="font-weight:bold">If you reply to this email, your message will be added to the discussion below:</div>
<a href="http://rabbitmq.1065348.n5.nabble.com/Long-timeout-if-server-host-becomes-unreachable-tp30275p30283.html" target="_blank" rel="nofollow" link="external">http://rabbitmq.1065348.n5.nabble.com/Long-timeout-if-server-host-becomes-unreachable-tp30275p30283.html</a>
</div>
<div style="color:rgb(102,102,102);font:11px/1.5em tahoma,geneva,helvetica,arial,sans-serif;margin-top:0.4em">
To unsubscribe from Long timeout if server host becomes unreachable, <a href="" target="_blank" rel="nofollow" link="external">click here</a>.<br>
<a href="http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml" rel="nofollow" style="font:9px serif" target="_blank" link="external">NAML</a>
</div></blockquote></div><br></div></div>
<br/><hr align="left" width="300" />
View this message in context: <a href="http://rabbitmq.1065348.n5.nabble.com/Long-timeout-if-server-host-becomes-unreachable-tp30275p30320.html">Re: Long timeout if server host becomes unreachable</a><br/>
Sent from the <a href="http://rabbitmq.1065348.n5.nabble.com/">RabbitMQ mailing list archive</a> at Nabble.com.<br/>