Matthias,<div><br></div><div>Thanks for the reply!<br><br><div class="gmail_quote">On Wed, Jul 29, 2009 at 02:02, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@lshift.net">matthias@lshift.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im"><br></div>
So the consumer has two connections - one to lb1 and one to mq1? That seems weird.<br>
<br>
Also, you say that mq1 has a connection *to* the consumer - are you implying that your load-balancing reverses the direction of connection establishment?<div class="im"></div></blockquote><div><br></div><div>I've done a bit more research on that, and it turns out our loadbalancer is performing NAT, so that what looks like a second connection is actually the other end of the connection established by the client.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- After ~5-10 minutes without messages published to the queue, the<br>
connection on the consumer goes away, and it establishes a new<br>
connection to lb1. mq1 at this point still has an established to<br>
connection to the consumer on the original port, in addition to the<br>
new connection. Messages published to the queue in question are now no<br>
longer delivered to the consumer.<br>
</blockquote>
<br></div>
Any idea what causes the original connection to get dropped?<br>
<br>
Also, what client are you using, and do you have heartbeats enabled on the AMQP connection?<div class="im"></div></blockquote><div><br></div><div>From what I've seen, the loadbalancer times out the connection, but fails to close it on mq1. Why this happens, I don't know, but a colleague mentioend he's seen LVS misbehaving like this in different setups before.</div>
<div><br></div><div>We're using this ruby library: <a href="http://github.com/tmm1/amqp/tree/master">http://github.com/tmm1/amqp/tree/master</a>, which implements amqp-0.8. We haven't got heartbeats enabled, and as far as I can see, the library doesn't expose a way to set it.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I am pretty sure some folks have used RabbitMQ behind a load balancer. Rabbit doesn't do anything fancy at the TCP/IP level, so generally this should work ok.</blockquote><div> </div><div>I am by now fairly convinced that this is a misconfiguration/bug with the loadbalancer per se. I will post more information as I get it.</div>
<div><br></div><div>Thanks for your help!</div><div>niko</div></div></div>