[rabbitmq-discuss] Rabbit connections dying from protected enviornment

Fri Oct 11 16:54:40 BST 2013

Hey all, I was hoping I could get some direction here that might help us out.  We've got a RabbitMQ production cluster up, and for the majority of our environments it works great.  However, we have a protected set of environments that from within it the connections keep dying.  If we go to the load balanced VIP, about 25 packets or so get exchanged, then there's a 200ms lull to acknowledge a packet from Rabbit, a quick resend/reack, and then nothing until the connection dies five minutes later.  If we try to go past the VIP straight to a node, it fails outright (we did update the firewall rules to point to the single node).  We've pulled the app out of this environment and it works fine outside, but within it seems to always exhibit this behavior.  I got one of our network guys to break down the route the packets have to take:

"Client Service is behind 2 stateful Cisco ASA firewalls.  They connect to a F5 LBed Rabbit MQ server that works fine when not connecting form the behind the FW.  Firewall ports open are TCP 5672-5673 and 15672.  Issue we having is related to the connection on TCP 5672.  Under working conditions, the client spins up 3 TCP connection to the server on 5672.  On the problem client, it spins up an initial 3 and immediately spins up additional connections.  Upon spinning up the additional connections it leaves the previous connections open, which are eventually reset after 5 min.

Thus the main issue is what is the client seeing that causing it to restart the connection.  The failed connections are very repeatable, it gets about 14 packets per side.  With the last packets being a ctag ### from the server, followed by a delayed ack from the client.

Did try adding some overrides for the protocol on the FWs that inline.  Added the below.

class-map RabbitMQ
description Disable connection timeouts and other for RabbitMQ
match port tcp eq 5672
policy-map global_policy
class RabbitMQ
  set connection random-sequence-number disable
  set connection timeout embryonic 0:00:20 tcp 0:00:00
"

Any ideas on what in that sequence could be hosing us up?

Thanks,
Kale
This communication contains information that is confidential,
proprietary in nature, and may also be attorney-client privileged
and/or work product privileged. It is for the exclusive use of the
intended recipient(s). If you are not the intended recipient(s) or
the person responsible for delivering it to the intended
recipient(s), please note that any form of dissemination,
distribution or copying of this communication is strictly
prohibited and may be unlawful. If you have received this
communication in error, please immediately notify the sender by replying
to this message and delete this email immediately. Thank you for your cooperation. 

Please be advised that neither Altegrity, its affiliates, its employees
or agents accept liability for any errors, omissions or damages
caused by delays of receipt or by any virus infection in this
message or its attachments, or which may otherwise arise as a
result of this e-mail transmission.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131011/c55bd16a/attachment.htm>