[rabbitmq-discuss] rabbitmq-discuss Digest, Vol 52, Issue 7

[Matthew Sackman]

On Wed, Sep 07, 2011 at 11:33:28AM -0700, Elias Levy wrote:
> > In the event that the entire cluster is stopped, due to mnesia, it's
> > normally required that the last node down comes up first - indeed the
> > other nodes in the cluster may very well refuse to start up (they'll
> > just block in the "starting database" step) until the nodes that were
> > alive when it died reappear.
> 
> Is there some timeout on this?  What if the old master is fully fried
> and won't be joining the cluster again?  Is there some sort of
> election within the remaining nodes to select a new master?  If so,
> how is the election resolved?  Slaves most closely synchronized to the
> old master wins?

Yeah, this is a bit of a problem. Essentially, yes, there is a timeout,
after which Rabbit will just give up trying to start. There are
definitely ways in which we could embue rabbitmqctl with the means to
tell rabbit to abandon all hope of some remote node ever rejoining (and
indeed, even ways to do this without the local rabbit coming up, which
is essential in this case), but we've not yet written this. There is a
bug open for this.

After that, the various local mnesia databases will try to merge
themselves back together, but if two nodes disagree about various
details and they can't delegate to a common node that was alive at the
point they failed, then mnesia will give up with an "unable to merge
schema" error, and there'll really be no hope at getting them both to
merge back together without resetting one of those nodes.

> > However, in catastrophic cases, such as instantaneous powerloss to the
> > whole cluster subsequent startup may well not be as orderly. In this
> > case, it's possible that the first node to recover that contained a
> > mirror of the queue, regardless of whether it was a master or slave,
> > takes the role of master. All other mirrors will then become slaves.
> 
> Presumably it waits for a while and then timeouts and becomes master, yes?

Erm possibly. Much of this ordering stuff comes from mnesia rather than
Rabbit. If the node's mnesia is happy to start up then Rabbit will then
restore queues without further delay. That may include the promotion of
slaves to master.

Matthew