[rabbitmq-discuss] RabbitMQ load balancing/failover with LVS

[Niko Felger]

Matthias,
Thanks for the reply!

On Wed, Jul 29, 2009 at 02:02, Matthias Radestock <matthias at lshift.net>wrote:

>
> So the consumer has two connections - one to lb1 and one to mq1? That seems
> weird.
>
> Also, you say that mq1 has a connection *to* the consumer - are you
> implying that your load-balancing reverses the direction of connection
> establishment?
>

I've done a bit more research on that, and it turns out our loadbalancer is
performing NAT, so that what looks like a second connection is actually the
other end of the connection established by the client.


> - After ~5-10 minutes without messages published to the queue, the
>> connection on the consumer goes away, and it establishes a new
>> connection to lb1. mq1 at this point still has an established to
>> connection to the consumer on the original port, in addition to the
>> new connection. Messages published to the queue in question are now no
>> longer delivered to the consumer.
>>
>
> Any idea what causes the original connection to get dropped?
>
> Also, what client are you using, and do you have heartbeats enabled on the
> AMQP connection?
>

>From what I've seen, the loadbalancer times out the connection, but fails to
close it on mq1. Why this happens, I don't know, but a colleague mentioend
he's seen LVS misbehaving like this in different setups before.

We're using this ruby library: http://github.com/tmm1/amqp/tree/master,
which implements
amqp-0.8. We haven't got heartbeats enabled, and as far as I can see, the
library doesn't expose a way to set it.


> I am pretty sure some folks have used RabbitMQ behind a load balancer.
> Rabbit doesn't do anything fancy at the TCP/IP level, so generally this
> should work ok.


I am by now fairly convinced that this is a misconfiguration/bug with the
loadbalancer per se. I will post more information as I get it.

Thanks for your help!
niko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090729/6aa7eb8b/attachment.htm