[rabbitmq-discuss] How to use federation with clustering for best HA-behaviour?

[Adam Aquilon]

Hello!

This is my first post, so be nice please? :-)

For the impatient:

I'm trying to understand how to achieve the best possible High
Availability-setup with RabbitMQ. The documentation seems to say
that I don't need to mirror the queues created by the federation
plugin, but then I don't get uninterrupted traffic.

Once I created a policy that included "ha-mode":"all" for federated
queues, I could crash single cluster nodes with no interruption in
traffic. Did I misunderstand the documentation or is there something
else going on here?


Background:

Currently, I have six test nodes running on my linux box, arranged
in three clusters of two nodes each: {n1,n2}, {n3,n4} and {n5,n6}.

The clusters are called "kista", "linkoping" and "lindholmen"
respectively. They are meant to represent different sites.
For the details, please see the end of this email which has
output from rabbitmqadmin.

I've defined separate upstreams and upstream-sets for each one-way
link between clusters. (Is that the way to do it?)

For test purposes I defined three topic exchanges per site:

  o "local.ci"  -- for intra-site messages
  o "ci"        -- for normal messages
  o "<site>.ci" -- filtered "ci" messages, federated

E.g. the "kista.ci" exchange only gets those "ci" messages we
would like to be visible to other sites. (Exchange to exchange
binding.)


During the test runs, I had a producer in "kista" and consumers
both in "kista" and the other two sites.


First test:

For the first test, I didn't have the "^federation:" policy rule.
In this case, it seemed as the federation plugin only setup one
link to each site, to one specific node.

When I deliberately killed that node, messages did not get sent
to the corresponding site.

When the node was restarted, the connection was resumed and messages
started flowing again.

Second test:

Here I defined the policy that mirrored the federation queues.

Now, when I killed one node, traffic continued uninterrupted (at
least seemingly). Restoring one node and killing the other also
worked (as expected). I had to kill both nodes at a site to stop
messages, which is the behaviour I wanted.


Question:

Have I understood correctly how to setup federation in clustered
sites?

Is there a better way?


Thank you for your time!
Kind regards,
Adam Aquilon


======================================================================

Policies for the "kista" site (nodes n1,n2):
+-------+------------+-------------------------------------------------------------+-----------------------+----------+
| vhost |    name    |                         definition                          |        pattern        | priority |
+-------+------------+-------------------------------------------------------------+-----------------------+----------+
| /     | lindholmen | {"ha-mode": "all", "federation-upstream-set": "lindholmen"} | ^lindholmen\.         | 0        |
| /     | linkoping  | {"ha-mode": "all", "federation-upstream-set": "linkoping"}  | ^linkoping\.          | 0        |
| /     | local      | {"ha-mode": "all"}                                          | ^(ci|local\.|kista\.) | 0        |
| /     | mirrored   | {"ha-mode": "all"}                                          | ^federation:          | 0        |
+-------+------------+-------------------------------------------------------------+-----------------------+----------+

Policies for the "linkoping" site (nodes n3,n4):
+-------+------------+-------------------------------------------------------------+--------------------------+----------+
| vhost |    name    |                         definition                          |         pattern          | priority |
+-------+------------+-------------------------------------------------------------+--------------------------+----------+
| /     | kista      | {"ha-mode": "all", "federation-upstream-set": "kista"}      | ^kista\.                 | 0        |
| /     | lindholmen | {"ha-mode": "all", "federation-upstream-set": "lindholmen"} | ^lindholmen\.            | 0        |
| /     | local      | {"ha-mode": "all"}                                          | ^(ci|local\.linkoping\.) | 0        |
| /     | mirrored   | {"ha-mode": "all"}                                          | ^federation:             | 0        |
+-------+------------+-------------------------------------------------------------+--------------------------+----------+

Policies for the "lindholmen" site (nodes n5,n6):
+-------+-----------+------------------------------------------------------------+----------------------------+----------+
| vhost |   name    |                         definition                         |          pattern           | priority |
+-------+-----------+------------------------------------------------------------+----------------------------+----------+
| /     | kista     | {"ha-mode": "all", "federation-upstream-set": "kista"}     | ^kista\.                   | 0        |
| /     | linkoping | {"ha-mode": "all", "federation-upstream-set": "linkoping"} | ^linkoping\.               | 0        |
| /     | local     | {"ha-mode": "all"}                                         | ^(ci|local\.|lindholmen\.) | 0        |
| /     | mirrored  | {"ha-mode": "all"}                                         | ^federation:               | 0        |
+-------+-----------+------------------------------------------------------------+----------------------------+----------+


Parameters for the "kista" site (nodes n1,n2):
+-------+----------------+-------------------------+--------------------------------------------------------------+
| vhost |      name      |        component        |                            value                             |
+-------+----------------+-------------------------+--------------------------------------------------------------+
| /     | lindholmen     | federation-upstream-set | [{"upstream": "lindholmen5"}, {"upstream": "lindholmen6"}]   |
| /     | lindholmen5    | federation-upstream     | {"uri": "amqp://localhost:5675", "max-hops": 1}              |
| /     | lindholmen6    | federation-upstream     | {"uri": "amqp://localhost:5676", "max-hops": 1}              |
| /     | linkoping      | federation-upstream-set | [{"upstream": "linkoping3"}, {"upstream": "linkoping4"}]     |
| /     | linkoping3     | federation-upstream     | {"uri": "amqp://localhost:5673", "max-hops": 1}              |
| /     | linkoping4     | federation-upstream     | {"uri": "amqp://localhost:5674", "max-hops": 1}              |
| /     | local-nodename | federation              | kista at myhost                                                |
| /     | local-username | federation              | guest                                                        |
+-------+----------------+-------------------------+--------------------------------------------------------------+

Parameters for the "linkoping" site (nodes n3,n4):
+-------+----------------+-------------------------+--------------------------------------------------------------+
| vhost |      name      |        component        |                            value                             |
+-------+----------------+-------------------------+--------------------------------------------------------------+
| /     | kista          | federation-upstream-set | [{"upstream": "kista1"}, {"upstream": "kista2"}]             |
| /     | kista1         | federation-upstream     | {"uri": "amqp://localhost:5671", "max-hops": 1}              |
| /     | kista2         | federation-upstream     | {"uri": "amqp://localhost:5672", "max-hops": 1}              |
| /     | lindholmen     | federation-upstream-set | [{"upstream": "lindholmen5"}, {"upstream": "lindholmen6"}]   |
| /     | lindholmen5    | federation-upstream     | {"uri": "amqp://localhost:5675", "max-hops": 1}              |
| /     | lindholmen6    | federation-upstream     | {"uri": "amqp://localhost:5676", "max-hops": 1}              |
| /     | local-nodename | federation              | linkoping at myhost                                          |
| /     | local-username | federation              | guest                                                        |
+-------+----------------+-------------------------+--------------------------------------------------------------+

Parameters for the "lindholmen" site (nodes n5,n6):
+-------+----------------+-------------------------+------------------------------------------------------------+
| vhost |      name      |        component        |                           value                            |
+-------+----------------+-------------------------+------------------------------------------------------------+
| /     | kista          | federation-upstream-set | [{"upstream": "kista1"}, {"upstream": "kista2"}]           |
| /     | kista1         | federation-upstream     | {"uri": "amqp://localhost:5671", "max-hops": 1}            |
| /     | kista2         | federation-upstream     | {"uri": "amqp://localhost:5672", "max-hops": 1}            |
| /     | linkoping      | federation-upstream-set | [{"upstream": "linkoping3"}, {"upstream": "linkoping4"}]   |
| /     | linkoping3     | federation-upstream     | {"uri": "amqp://localhost:5673", "max-hops": 1}            |
| /     | linkoping4     | federation-upstream     | {"uri": "amqp://localhost:5674", "max-hops": 1}            |
| /     | local-nodename | federation              | lindholmen at myhost                                       |
| /     | local-username | federation              | guest                                                      |
+-------+----------------+-------------------------+------------------------------------------------------------+