[rabbitmq-discuss] RabbitMQ production setup questions around clustering

[Aaron Westendorf]

Dave,

Including the list on the reply.

On Wed, Jul 21, 2010 at 5:44 PM, Dave Greggory <davegreggory at yahoo.com> wrote:
>
> Your postal service analogy is good one and that is how I understood it mostly.
> From what I understood, if a client requests a message from a queue that is not
> on the node he has the current connection to, because of clustering, you will
> get the message back indirectly. But your analogy makes it sound like the client
> will actually need to establish a second connection to the node that has the
> queue. I'm fine with that, that's not a big deal.

I may be wrong in this, Alexandru hinted that clustering might be more
of flexible than I described.

> I declare my (durable) queues from the both producer and consumer ends because
> exchanges would just drop the messages if we left it up to consumers and no
> consumers were up at the time a producer sends message. So But I'm trying to
> understand where a queue will be created when it's  declared (will it be in the
> node the client has the connection open to or  could it end up existing in
> another node)?

We have run in to that exact chicken-and-egg problem.  We don't have a
need for persistent messages before a consumer starts for the first
time, so we've been able to allow the consumer to drive the queue
declare and drop unrouteable messages.  If that doesn't work for you,
then perhaps you can have something that sits outside your apps and
ensures that all queues are declared and in good health.  We've taken
to adding a several monitoring tools because it's so easy and it adds
to our reliability.

The queue will exist on the host to which you are connected when you declare it.

> I may have misused the term load-balancer in this context (just because that's
> what we use that piece of hardware most often). Let's just call it node-switcher
> for this purpose. I intended it to serve two purposes,
> a) Hide the hostnames/ip-addresses of rabbitmq nodes from producers and clients
> b) Serve as switch for determining which rabbitmq nodes consumers/producers
> connect to, so that we can:
>     i. Switch all new consumer/producer connections  automatically to the other
> node if the first node goes down
>
>     ii. Upgrade rabbitmq nodes without affecting producer/consumer nodes
> because we can switch out different rabbitmq nodes

We've solved this problem with DNS aliases.  It's not automated by
design but it does allow us to cleanly disconnect nodes and have the
clients reconnect to another.  The trick there is that if a queue
existed on a node that is shutdown, the clients have to also declare
it, its bindings and the consumers again.  This was best accomplished
for us by simply killing the clients off on a Rabbit disconnect and
letting the "hypervisor" restart them.

That only worked because all consumers of the queue were connected to
the same host on which the queue resided.  If Alexandru is correct and
the cluster can be treated as a pool, then the reconnect scenario has
to include those other clients re-initializing their consumer calls to
the new queue, after it is re-declared.  So perhaps it is best to
always connect to the node hosting the queue you want to consume from.

cheers,
Aaron

-- 
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aaron at agoragames.com
www.agoragames.com