[rabbitmq-discuss] RabbitMQ Clustering with Federation

[Simon MacMullen]

On 31/07/2013 5:35PM, Robert Parker wrote:
> I'm having some trouble getting the right architecture configured to
> support rabbitmq federation in a clustered high availability
> environment.  Our basic setup is four clusters of two nodes each which
> are separated by a WAN.
>
> "home" is the "home" cluster and "remote1", "remote2" and "remote3" are
> remote clusters.  There are a lot of queues being created and published
> to via the API identically on all clusters but I only want to federate
> ONE of the queues, which is called "EventListener" back to the "home"
> cluster from all the "remote" clusters.  I want messages published to
> this queue on each remote cluster to be federated to the home cluster to
> be consumed by a separate event listener service that only lives at the
> "home" site.

Just to be clear: you are aware that (currently) it's only *exchanges* 
which can be federated, not queues?

> Each cluster is configured with:

<snip>

> We have configured three hardware load balancer VIPs that load balance
> connections on port 5672 between both nodes in each cluster, and have
> configured federation upstreams to use these and also configured our
> application to use these VIPs for queue/exchange creation and for
> publishing messages.  Here's our federation config script thats run on
> node1 of the home cluster only:

<snip>

> However I am noticing that the "rabbitmqctl eval
> 'rabbit_federation_status:status()." command only shows federation
> config on node2 of the home cluster despite having been configured on node1:

Federation works by establishing link(s) from the downstream to the 
upstream(s). You can define the upstreams on any node in the downstream 
cluster (the configuration is cluster-wide) and the cluster will 
arbitrarily start the link processes on any node (and will migrate them 
around the cluster if nodes go down).

*However*, rabbit_federation_status:status() will only report links that 
are running on the local node. This is probably not well documented. 
Hmm. Let's fix that.

(The HTTP API exported by rabbitmq_federation_management will report 
links on all nodes.)

<snip>

> And I'm noticing that messages published to the queues via the load
> balancer VIPs to the remote clusters are not showing up in the
> "EventListener" queue of the home cluster.

Note that you have federated an exchange called "EventListener", not a 
queue. So if you bind a queue to the exchange "EventListener" on home, 
and publish to the exchange "EventListener" on remote1/2/3 then you 
should see messages turning up in the queue on home. Is that what you're 
doing?

> Is this not the right way to set this up?

It sounds like you're basically right, apart from the exchange vs queue 
thing.

> Is it possible to configure
> this without using an external load balancer and still achieve high
> availability?

Since 3.1.0, yes. You can specify multiple URIs in a single upstream, 
and federation will connect to a randomly chosen one (and keep retrying 
different ones if that one goes down). Use something like:

rabbitmqctl set_parameter federation-upstream remote1-upstream 
'{"uri":["amqp://remote1-node1","amqp://remote1-node2"],"expires":3600000}'

> If I federate to a single node of a cluster directly,
> what happens if that node goes down?

If you only specify one URI, it will keep retrying until that node comes 
back up.

> Do I have to configure redundant
> federation between every node of home cluster to every node of each
> remote cluster?

No - since the config is shared across the cluster. So you can set up 
the upstreams on one node of the home cluster, pointing to all nodes of 
the remote clusters.

> If I don't use a load balancer VIP to publish to from my
> application, if the single node of the cluster I've configured goes
> down, it wont help me that the other node is still available.

Not sure what you mean here. Hopefully I've explained enough now.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal