Actually, why couldn't RabbitMQ use a majority quorum for determining who's the most up-to-date guy?<div><br></div><div>Assume 3 nodes: A, B, C. Assume that A is the leader.</div><div><br></div><div>Stop A: now B and C elect a leader and, say, they elect B.</div>
<div>Stop B: now C knows it's no longer part of a quorum and it just sits there unresponsive. B doesn't behave like a leader either (if it was just partitioned and not killed). Ok.</div><div>Start A: A and C now form a quorum with C as the most up-to-date member; they elect C as the leader and A synchronizes from C.</div>
<div>Start B: B synchronizes from C too.</div><div><br></div><div>This seems implementable and I believe that's what replicated databases like Galera do; is it just difficult, or is there a theoretical issue related to RabbitMQ specifically, or are you ruling out this option because it requires at least 3 nodes to actually be H/A?</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Nov 15, 2012 at 5:28 PM, Eugene Kirpichov <span dir="ltr"><<a href="mailto:ekirpichov@gmail.com" target="_blank">ekirpichov@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Simon,<div><br></div><div>Thank you, it all makes sense now.</div><div><br></div><div>So, we can say "either reboot one node at a time, or - if you're rebooting all of them - make sure they start in reverse order, or simultaneously in a window of 30sec max".</div>
<div><br></div><div>Can we also say "if something bad happened, kill -9 all rabbits, then start them in a window of 30sec max"? [I'm talking kill -9 because in some cases with messed up startup order, rabbitmqctl stop also hangs]</div>
<div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">On Thu, Nov 15, 2012 at 4:33 PM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On 15/11/12 12:04, Eugene Kirpichov wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Is RabbitMQ HA and clustering sufficiently reliable to use it in<br>
scenarios where the network is good, but nodes can reboot at any time?<br>
</blockquote>
<br></div>
We believe so.<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
My understanding was that this is what "HA" is supposed to mean, but<br>
then I read this:<br>
<br>
<a href="http://stackoverflow.com/questions/8654053/rabbitmq-cluster-is-not-reconnecting-after-network-failure" target="_blank">http://stackoverflow.com/<u></u>questions/8654053/rabbitmq-<u></u>cluster-is-not-reconnecting-<u></u>after-network-failure</a><br>
</blockquote>
<br></div>
This one was a network partition - clusters don't handle partitions well.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<a href="http://rabbitmq.1065348.n5.nabble.com/Cluster-nodes-stop-start-order-can-lead-to-failures-td21965.html" target="_blank">http://rabbitmq.1065348.n5.<u></u>nabble.com/Cluster-nodes-stop-<u></u>start-order-can-lead-to-<u></u>failures-td21965.html</a><br>
</blockquote>
<br>
This one is the stop-start ordering problem (discussed below).<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<a href="http://rabbitmq.1065348.n5.nabble.com/Cluster-busting-shut-off-all-nodes-at-the-same-time-td22971.html" target="_blank">http://rabbitmq.1065348.n5.<u></u>nabble.com/Cluster-busting-<u></u>shut-off-all-nodes-at-the-<u></u>same-time-td22971.html</a>:<br>
</blockquote>
<br>
As was this.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<a href="http://rabbitmq.1065348.n5.nabble.com/Repairing-a-a-crashed-cluster-td22466.html" target="_blank">http://rabbitmq.1065348.n5.<u></u>nabble.com/Repairing-a-a-<u></u>crashed-cluster-td22466.html</a><br>
</blockquote>
<br>
This one was unclear ("something happened"), but I took the question to be about removing a node from a cluster when that node cannot come up. This is handled badly in 2.x, but 3.0 will have a rabbitmqctl subcommand to do that.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<a href="http://grokbase.com/t/rabbitmq/rabbitmq-discuss/125nxzf5nh/highly-available-cluster" target="_blank">http://grokbase.com/t/<u></u>rabbitmq/rabbitmq-discuss/<u></u>125nxzf5nh/highly-available-<u></u>cluster</a><br>
</blockquote>
<br>
This is another stop-start ordering problem.<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
And now I'm not so sure. It seems that there are a lot of scenarios<br>
where merely rebooting the nodes in some order brings the cluster into a<br>
state from which there is no automatic way out.<br>
</blockquote>
<br></div>
So the most common problem you cited above looks like this (let's suppose we have a two node cluster AB for simplicity):<br>
<br>
1) Stop B<br>
2) Stop A<br>
3) Start B<br>
4) Start A<br>
<br>
3) will fail. More precisely, it will wait for 30 seconds to see if 4) happens, and if not then it will fail.<br>
<br>
Why? Well, a lot could have happened between 1) and 2). You could have declared or deleted all sorts of queues, changed everybody's password, all sorts of things. B has no way to know; it was down.<br>
<br>
It *can't* (responsibly) start up by itself. So it has to wait around for A to become available.<br>
<br>
To be more general, the last node to be stopped has to be the first one to be started. No other node knows what's happened in the mean time!<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Questions:<br>
1) Is there a set of assumptions or procedures under which I can be<br>
*certain* that my RabbitMQ cluster will actually tolerate unexpected<br>
node failures? Maybe something like "no more than 1 node down at the<br>
same time", or "at least X seconds between reboots", or "after a node<br>
reboots, restart all rabbit instances" or "have at most 2 nodes" etc.?<br>
I'm asking because I need to at least document this to my customers.<br>
</blockquote>
<br></div>
* Avoid network partitions. You can recover (see <a href="http://next.rabbitmq.com/partitions.html" target="_blank">http://next.rabbitmq.com/<u></u>partitions.html</a>) but it's a good way to introduce problems.<br>
<br>
* If you stop all nodes, the first (disc) node to start should be the last one to stop.<br>
<br>
* If you have RAM nodes, start them after you've started some disc nodes.<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2) To what degree are the issues described in those threads fixed in the<br>
next release of RabbitMQ - 3.0.0, and how soon is it expected to be<br>
production-ready?<br>
</blockquote>
<br></div>
3.0.0 will not remove this stop-start ordering constraint. I don't see how anything can.<br>
<br>
However, it will have some enhancements to make clustering problems easier to detect and fix (such as a removing a dead node without its cooperation, making sure you don't get into a state where nodes disagree on whether they are clustered with each other) and it will also detect and warn more clearly about network partitions.<br>
<br>
It should be available any day now.<br>
<br>
Cheers, Simon<span><font color="#888888"><br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div></div></div><div class="im">-- <br>Eugene Kirpichov<br><a href="http://www.linkedin.com/in/eugenekirpichov" target="_blank">http://www.linkedin.com/in/eugenekirpichov</a><br>
We're hiring! <a href="http://tinyurl.com/mirantis-openstack-engineer" target="_blank">http://tinyurl.com/mirantis-openstack-engineer</a><br>
</div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Eugene Kirpichov<br><a href="http://www.linkedin.com/in/eugenekirpichov" target="_blank">http://www.linkedin.com/in/eugenekirpichov</a><br>We're hiring! <a href="http://tinyurl.com/mirantis-openstack-engineer" target="_blank">http://tinyurl.com/mirantis-openstack-engineer</a><br>
</div>