<div dir="ltr">Hi Roy,<div><br></div><div style>I am not using time.sleep in my app.</div><div style><br></div><div style>Worker consumes message from the queue. It then starts processing it which takes several hours to complete. In the meantime, the connection is dropped off due to timeout. Once the processing is complete, the worker tries to send ACK. While doing so, it fails with the error below and the worker shuts down.</div>
<div style><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div style><div style>render_jobs_consumer - INFO - tableau-classic-admin0-borders-en-us-extent-101.mbtiles published for merging</div>
</div><div style><div style>render_jobs_consumer - INFO - tableau-classic-admin0-borders-en-us-extent-101.mbtiles rendering completed in 334.802966833. Sending acknowledgement</div></div><div style><div style><div style>pika.adapters.base_connection - ERROR - Read empty data, calling disconnect</div>
</div></div><div style><div style><div style>pika.adapters.base_connection - WARNING - Unknown state on disconnect: 5</div></div></div><div style><div style><div style>pika.connection - WARNING - Disconnected from RabbitMQ at <a href="http://192.168.2.182:5672">192.168.2.182:5672</a> (0): </div>
</div></div><div style><div style><div style>pika.adapters.select_connection - INFO - Unregistering poller on fd 3</div></div></div></blockquote><div style><div><br></div><div style>Thanks</div><div style>Nishanth</div></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Mar 8, 2013 at 11:44 AM, Gavin M. Roy <span dir="ltr"><<a href="mailto:gmr@meetme.com" target="_blank">gmr@meetme.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
Depends on your code. Do you use time.sleep in your app? If you enable logging in your app, set pika to INFO -- do you see any messages on the Pika side? Are you consuming messages when it times out?</div>
<div class="HOEnZb"><div class="h5"><div><br></div>
<div></div>
<p style="color:#a0a0a8">On Friday, March 8, 2013 at 1:13 PM, Nishanth Babu Achuthan wrote:</p>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px">
<span><div><div><div dir="ltr">Hi Michael & Emile,<div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>Thank you very much for the inputs.</div><div><br></div><div>
I set the heartbeat interval to 25 seconds as below in the client.</div>
<div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>parameters = pika.ConnectionParameters(host=RABBITMQ_HOSTNAME,heartbeat_interval=25,credentials=pika.PlainCredentials(RABBITMQ_USERNAME,RABBITMQ_PASSWORD))</div>
</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>connection = pika.SelectConnection(parameters,on_connected)</div></div></blockquote><div><div>
<br></div></div><div><div>My rabbitmq conifg looks like below</div></div><div><div><br></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div>
[</div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div> {rabbit,</div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div><div><div> [</div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div> {heartbeat,25},</div>
</div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div> {tcp_listen_options, [binary, {packet, raw},</div></div>
</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div> {reuseaddr, true},</div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div><div><div> {backlog, 128},</div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div><div><div> {nodelay, true},</div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>
<div> {keepalive, true}]}</div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div> ]</div>
</div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div> }</div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div><div><div>].</div></div></div></blockquote><div><div><div><br></div></div></div><div><div><div>I ran the rabbitmqctl list_connections timeout as below</div></div>
</div><div><div><div><br></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div>root@ip-10-78-237-240:/mnt# rabbitmqctl list_connections timeout</div>
</div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div>Listing connections ...</div></div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div><div><div><div>25</div></div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div>...done.</div></div></div>
</div></blockquote><div><div><div><div><br></div></div></div></div><div><div><div><div>After 25 seconds, the connection dropped off. It says heartbeat_timeout in the logs</div>
</div></div></div><div><div><div><div><br></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div>=INFO REPORT==== 8-Mar-2013::17:59:03 ===</div>
</div></div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div>accepting AMQP connection <0.542.0> (<a href="http://10.79.18.130:60527" target="_blank">10.79.18.130:60527</a> -> <a href="http://10.78.237.240:5672" target="_blank">10.78.237.240:5672</a>)</div>
</div></div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><br></div></div></div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div><div><div><div><div>=ERROR REPORT==== 8-Mar-2013::18:00:18 ===</div></div></div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>
<div><div><div>closing AMQP connection <0.542.0> (<a href="http://10.79.18.130:60527" target="_blank">10.79.18.130:60527</a> -> <a href="http://10.78.237.240:5672" target="_blank">10.78.237.240:5672</a>):</div></div>
</div></div>
</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div>{heartbeat_timeout,running}</div></div></div></div></div></blockquote><div>
<div><div><div><div><br></div></div></div></div></div><div><div><div><div><br></div></div></div></div><div><div><div><div>I am not sure why the connection is dropped here. Am I doing anything wrong?</div>
</div></div></div></blockquote><div><div><br></div><div>Thanks</div><div>Nishanth</div></div></div><div><br><br><div>On Fri, Mar 8, 2013 at 7:50 AM, Laing, Michael P. <span dir="ltr"><<a href="mailto:Michael.Laing@nytimes.com" target="_blank">Michael.Laing@nytimes.com</a>></span> wrote:<br>
<blockquote type="cite"><div>We keep connections alive for long periods in EC2.<br>
<br>
We set the heartbeat to 25 secs to achieve this, as elastic load<br>
balancers, in particular, have short timeouts.<br>
<br>
I would recommend upgrading to the latest rabbitmq and setting the<br>
heartbeat in the configuration.<br>
<br>
Look to see that it shows up in the management console. It's possible for<br>
client interface code to not agree to heartbeat w the server.<br>
<br>
Michael<br>
<div><div><br>
On 3/8/13 4:56 AM, "Emile Joubert" <<a href="mailto:emile@rabbitmq.com" target="_blank">emile@rabbitmq.com</a>> wrote:<br>
<br>
><br>
>Hi,<br>
><br>
>On 07/03/13 22:59, Nishanth wrote:<br>
><br>
>> While they process the message, rabbitmq drops their connection silently<br>
>> as it considers them to have been idle for a long time.<br>
><br>
>The broker will not drop connections silently. The culprit is more<br>
>likely a firewall and enabling heartbeats is the correct way to solve<br>
>the problem.<br>
><br>
>> I tried to set up heartbeat interval in worker connections but it didn't<br>
>> resolve the issue. It keeps timing out. I am not able to understand how<br>
>> this heartbeat work or how to debug to find out why its timing out.<br>
><br>
>You can check that the heartbeat is indeed recognised in the broker by<br>
>issuing "rabbitmqctl list_connections timeout".<br>
><br>
>If the heartbeat is set too long then connections may of course still be<br>
>dropped, e.g. if an intervening firewall has a 10 minute timeout and the<br>
>AMQP heartbeats are exchanged every 15 minutes then connections will<br>
>still be dropped.<br>
><br>
><br>
><br>
>-Emile<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
</div></div>>_______________________________________________<br>
>rabbitmq-discuss mailing list<br>
><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></blockquote></div><br></div>
</div><div><div>_______________________________________________</div><div>rabbitmq-discuss mailing list</div><div><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a></div>
<div><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>
</div></div><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>