[rabbitmq-discuss] Keeping idle connections alive in EC2

[Nishanth Babu Achuthan]

Hi Roy,

I am not using time.sleep in my app.

Worker consumes message from the queue. It then starts processing it which
takes several hours to complete. In the meantime, the connection is dropped
off due to timeout. Once the processing is complete, the worker tries to
send ACK. While doing so, it fails with the error below and the worker
shuts down.

render_jobs_consumer - INFO -
tableau-classic-admin0-borders-en-us-extent-101.mbtiles published for
merging
render_jobs_consumer - INFO -
tableau-classic-admin0-borders-en-us-extent-101.mbtiles rendering completed
in 334.802966833. Sending acknowledgement
pika.adapters.base_connection - ERROR - Read empty data, calling disconnect
pika.adapters.base_connection - WARNING - Unknown state on disconnect: 5
pika.connection - WARNING - Disconnected from RabbitMQ at
192.168.2.182:5672(0):
pika.adapters.select_connection - INFO - Unregistering poller on fd 3


Thanks
Nishanth


On Fri, Mar 8, 2013 at 11:44 AM, Gavin M. Roy <gmr at meetme.com> wrote:

>  Depends on your code. Do you use time.sleep in your app? If you enable
> logging in your app, set pika to INFO -- do you see any messages on the
> Pika side? Are you consuming messages when it times out?
>
>  On Friday, March 8, 2013 at 1:13 PM, Nishanth Babu Achuthan wrote:
>
> Hi Michael & Emile,
>
> Thank you very much for the inputs.
>
> I set the heartbeat interval to 25 seconds as below in the client.
>
> parameters      =
> pika.ConnectionParameters(host=RABBITMQ_HOSTNAME,heartbeat_interval=25,credentials=pika.PlainCredentials(RABBITMQ_USERNAME,RABBITMQ_PASSWORD))
>
> connection      =       pika.SelectConnection(parameters,on_connected)
>
>
> My rabbitmq conifg looks like below
>
> [
>
>         {rabbit,
>
>                 [
>
>                         {heartbeat,25},
>
>                         {tcp_listen_options,    [binary,        {packet,
>      raw},
>
>                                                 {reuseaddr,     true},
>
>                                                 {backlog,       128},
>
>                                                 {nodelay,       true},
>
>                                                 {keepalive,     true}]}
>
>                 ]
>
>         }
>
> ].
>
>
> I ran the rabbitmqctl list_connections timeout as below
>
> root at ip-10-78-237-240:/mnt# rabbitmqctl list_connections timeout
>
> Listing connections ...
>
> 25
>
> ...done.
>
>
> After 25 seconds, the connection dropped off. It says heartbeat_timeout in
> the logs
>
> =INFO REPORT==== 8-Mar-2013::17:59:03 ===
>
> accepting AMQP connection <0.542.0> (10.79.18.130:60527 ->
> 10.78.237.240:5672)
>
>
> =ERROR REPORT==== 8-Mar-2013::18:00:18 ===
>
> closing AMQP connection <0.542.0> (10.79.18.130:60527 ->
> 10.78.237.240:5672):
>
> {heartbeat_timeout,running}
>
>
>
> I am not sure why the connection is dropped here. Am I doing anything
> wrong?
>
>
> Thanks
> Nishanth
>
>
> On Fri, Mar 8, 2013 at 7:50 AM, Laing, Michael P. <
> Michael.Laing at nytimes.com> wrote:
>
> We keep connections alive for long periods in EC2.
>
> We set the heartbeat to 25 secs to achieve this, as elastic load
> balancers, in particular, have short timeouts.
>
> I would recommend upgrading to the latest rabbitmq and setting the
> heartbeat in the configuration.
>
> Look to see that it shows up in the management console. It's possible for
> client interface code to not agree to heartbeat w the server.
>
> Michael
>
> On 3/8/13 4:56 AM, "Emile Joubert" <emile at rabbitmq.com> wrote:
>
> >
> >Hi,
> >
> >On 07/03/13 22:59, Nishanth wrote:
> >
> >> While they process the message, rabbitmq drops their connection silently
> >> as it considers them to have been idle for a long time.
> >
> >The broker will not drop connections silently. The culprit is more
> >likely a firewall and enabling heartbeats is the correct way to solve
> >the problem.
> >
> >> I tried to set up heartbeat interval in worker connections but it didn't
> >> resolve the issue. It keeps timing out. I am not able to understand how
> >> this heartbeat work or how to debug to find out why its timing out.
> >
> >You can check that the heartbeat is indeed recognised in the broker by
> >issuing "rabbitmqctl list_connections timeout".
> >
> >If the heartbeat is set too long then connections may of course still be
> >dropped, e.g. if an intervening firewall has a 10 minute timeout and the
> >AMQP heartbeats are exchanged every 15 minutes then connections will
> >still be dropped.
> >
> >
> >
> >-Emile
> >
> >
> >
> >
> >
> >
> >
> >
> >_______________________________________________
> >rabbitmq-discuss mailing list
> >rabbitmq-discuss at lists.rabbitmq.com
> >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130308/8f227308/attachment.htm>