[rabbitmq-discuss] BIGIP/F5 and Connection resets

[Jason McIntosh]

I saw that heartbeats are turned on with 3.0.0 - the question is does 2.8.7
have heartbeat support by default on the server side or can it be turned
on?  How is this different the SO_KEEPALIVE:
http://www.erlang.org/doc/man/inet.html

 There's no details on the documentation about configuring the heartbeat on
2.8.7 setting if there is a configuration option for it.  I did find this:
https://bugs.launchpad.net/nova/+bug/856764
AND
http://rabbitmq.1065348.n5.nabble.com/Timeout-for-hung-connection-on-rabbitmq-server-td21354.html
I'm not sure what the difference is between a rabbit heartbeat and erlangs
keepalive?  Or is heartbeat just turning on keepalive?  I'm going to look
at the erlang configuration as a possible fix.  We're trying to do this on
the server side so we don't have to modify all of our clients.

Here's a few more details on the exact issue we've encountered
RAB_SERVER <-> F5/Firewall/etc. <-> RAB_CLIENT
the client and server don't have any traffic going through so they get
disconnected in one of two ways.  If we used just a router and not a load
balancer it seems most network firewalls auto disconnect with NO
notification both clients after an hour of no TCP/IP traffic (from what I
understand, SSH for example uses a TCP/IP Keepalive to keep their
connections going).  The load balancer on the other hand does a graceful
RESET to both client and server after our delay of 300 seconds.  We've
tried increasing the delay, but it looks like 2.8.7 by default doesn't set
any kind of heartbeat or keepalive so even if we set it to an hour, we'd
still get resets.

Rabbit clients using spring & the shovel configuration itself handle resets
of the connection pretty gracefully.  The issue is I'd rather not have
resets as I don't trust all clients to be nice about reconnecting.  The
spring code reconnects gracefully - the standard rabbit client doesn't seem
to be as nice about reconnecting.

Jason






On Tue, Dec 18, 2012 at 6:52 AM, Emile Joubert <emile at rabbitmq.com> wrote:

>
> Hi Jason,
>
> On 17/12/12 22:53, Jason McIntosh wrote:
>
> > From what I've been told, most network devices drop TCP/IP
> > connections after an hour automatically (i.e. switches, routers,
> > etc.) and force a reconnect - not sure if Rabbit handles this?
>
> The broker handles a closed connection by logging the event.
>
> > There is a work around at the moment we're looking into.  It looks like
> > most (if not all) Rabbit interfaces have a heartbeat setting that would
> > allow connections to be maintained
>
> Yes, AMQP heartbeats are a suitable solution if intermediate network
> devices are closing idle connections too eagerly. Heartbeats are enabled
> by default in v3.0.0 if the client supports them.
>
> Another options is to write client applications to re-establish closed
> connections after a suitable delay.
>
> If consumers receive messages asynchronously instead of polling the
> broker for messages, and there are no messages to deliver then the TCP
> connection will carry no traffic.
>
> > Other questions related to the above:  Is the AMQP protocol designed to
> > have the clients just sit idle, no communication with the servers until
> > a message comes in
>
> Rabbit does not impose limits on connection duration and idle
> connections are permitted to remain so indefinitely.
>
> > If there is no check, and the client hasn't received a message in a
> > day at what point do you retry your connection to verify you're even
> > still connected? It sounds like in this case you could have a
> > consumer of a queue lose it's connection and never realize
>
> If connections are being severed silently without notification from the
> network stack then you must use AMQP heartbeats so that both sides of
> the connection become aware of it. Otherwise the network stack should
> provide notification that the connection has been interrupted.
>
> > Then you'd have a server just sitting backlogging messages waiting
> > for a client (or worse depending on configuration, just dropping all
> > the messages).
>
> The broker will not drop messages due to broken connections and the
> protocol is designed so that a guarantee of message delivery can be
> offered even in the face of an unreliable transport.
>
> > Looked at the AMQP 0-9-1 specifications and not seeing a whole lot
> > defined on how AMQP implementers are supposed to work with TCP/IP
> > traffic (other than some heartbeat definitions).
>
> The "connection" class relates to the transport, which must be
> stream-oriented.
>
>
> -Emile
>
>
>
>
>
>
>
>
>
>
>


-- 
Jason McIntosh
http://mcintosh.poetshome.com/blog/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121219/7c8d0179/attachment.htm>