[rabbitmq-discuss] rabbitmq cluster manage/ops questions

[Francesco Mazzoli]

Hi Torin,

At Wed, 15 Aug 2012 12:33:08 -0700,
Torin Sandall wrote:
> I'm working on a project involving clustered rabbitmq brokers and I would like
> to gain a better understanding of the operational constraints. I've read the
> clustering article on the site, but I feel like I don't have a solid
> understanding of it yet.
> 
> Specific questions:
> 
> 1) What constraints need to be observed to guarantee the cluster state remains
> consistent. I.e., the cluster will not fall into a "split brain" state.

There are various situations that will mess up clustering.  In general, problems
arise when tables change on some node while other cluster nodes are unaware of
the change.  When something like that happen, mnesia will spit out a "fatal"
error and rabbit will die.

Common scenarios:

  1. A and B are in a cluster.  The connection between A and B is severed, and
     some changes are made to the tables in A or/and B.  When the connection is
     re-established, mnesia will detect that the tables have diverged and die.
  2. A and B are in a cluster.  A is stopped.  B is stopped, reset, and brought
     back online.  A is brought back online, it tries to cluster with B but B's
     schema is now incompatible with A's -> A dies.
  3. A gets clustered to B as a ram node.  C gets clustered to B as a disc
     node. A is stopped.  B is stopped, reset and started again.  A is stopped
     and started again.  A dies because it still thinks that B is in the
     cluster, given that it is a ram node.

We can't do much about 1.

2 and 3 have many variations, and we are working to make the situation much
better - the changes should be available in the next feature release.

> 2) Is there anything invalid or problematic about the tests I describe below?
> 
> I've been running tests involving 2-3 clustered brokers. All brokers are
> running 2.7.1. The cookies are synched properly.

Please upgrade to the last version :).

> The cluster can be in one of the following states:
> 
> 2 disc
> 2 disc, 1 ram
> 3 disc
> 
> In any of the states, my test can kill one of the brokers (ram or disc, it
> doesn't discriminate.) If a broker is killed the next event the test would
> execute is either a restart of the dead broker or a replacement of the dead
> broker. Replacement is done by deleting the mnesia database on that node and
> then service start, stop_app, reset, force_cluster, start_app.

Mhm.  Using `force_cluster' is almost always a bad idea.  However in the current
state of things it is necessary sometimes.  When the mentioned clustering
changes will be released `force_cluster' will be removed, since there will be no
reasonable use cases.

For example your use case is a bad one, since `force_cluster' relies on the node
you're clustering to already have tables matching the other nodes', but that
will not be the case since you have just reset it.

In general make sure that you know what you're doing when you're using it.

> In the 2 broker-cluster states, the cluster can be grown to size of 3.
> 
> In the 3 broker-cluster states, the cluster can be shrunk to size of 2 with a
> constraint that it won't ever shrink to 1 disc/1 ram.
> 
> Growing and shrinking of the cluster is always done by running rabbitmqctl
> commands. I.e., there's no cluster configuration in the rabbitmq.config
> file. For those who will ask, the commands I'm running to grow and shrink the
> cluster are:
> 
> 1) grow cluster by adding a ram node
> rabbitmqctl stop_app
> rabbitmqctl reset
> rabbitmqctl cluster <existing-disc-node>
> rabbitmqctl start_app
> 
> 2) grow cluster by adding a disc node
> rabbitmqctl stop_app
> rabbitmqctl reset
> rabbitmqctl cluster <existing-disc-node> <node-to-be-added>
> rabbitmqctl stop_app
> 
> 3) shrink cluster by removing a node
> rabbitmqctl stop_app
> rabbitmqctl reset
> rabbitmqctl start_app

These all look fine.

> I'm not inserting any delays between execution of events (other than the
> implicit delay of having to ssh into the server and execute the command.)
> 
> One issue I've encountered so far:
> 
> 1) rabbit fails to start after shrinking 2 disc/1 ram cluster to 2 disc
> cluster and then killing a disc node. Here's teh log from the disc node which
> fails to start. There's also output from my test script at the bottom which
> shows the cluster status: http://pastebin.com/95McUzkb

Uhm.  Could you paste the test code or what you're doing in the test in detail?
My suspicion is that you're using `force_reset' as you described above, and thus
the schema tables don't match.

--
Francesco * Often in error, never in doubt