<div dir="ltr">So to confirm my understanding here. Message 1 was synched to both queues before I stopped the disc node. So when I shut down the disc node, the ram node become the "master". As such, it should have had message 1 and 2. When I brought up the disc node it is a slave and had it's messages essentially reset. Per the docs: "<span style="color:rgb(85,85,85);font-family:Verdana,sans-serif;font-size:13px;line-height:18px">As such, when a slave rejoins a mirrored-queue, it throws away any durable local contents it already has and starts empty.". </span> So when I now publish message 2, that goes to the ram node, which is now the master, and we have 2 messages total (disc node still off). <div>
<br></div><div style>The question becomes then - is there any way to recover the messages the RAM node had if the disc node comes back and the ram node subsequently failed? Granted this is an unlikely situation, but I'm trying to plan for recover scenarios here :) Do I need to remove the HA policy, then bring up the RAM node to get the RAM node messages (since it was master prior to it's restart)? Since I can't shut off the disc node in this case, just trying to figure out how to recover. I can see taking this a step further - if I had 3 nodes in a cluster, and 1 node went down, but the other 2 nodes each got an even distribution of the queues... hrmm, would make it somewhat interesting to recover. I'd almost have to shutdown yet one more node, start up the original node, make sure that messages get consumed, then start the third node it sounds like. </div>
<div style><br></div><div style>I've started testing on the new 3.1 nightly - I'm hoping it makes this a lot cleaner!<br></div><div style>Thanks!<br>Jason</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Fri, Apr 26, 2013 at 3:44 AM, Emile Joubert <span dir="ltr"><<a href="mailto:emile@rabbitmq.com" target="_blank">emile@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Hi Jason,<br>
<div class="im"><br>
On 25/04/13 21:09, Jason McIntosh wrote:<br>
> I have 2 nodes in an rabbit cluster, one disc, one ram node. I'm seeing<br>
> messages get lost<br>
<br>
</div>The explanation can be found here:<br>
<a href="http://www.rabbitmq.com/ha.html#behaviour" target="_blank">http://www.rabbitmq.com/ha.html#behaviour</a><br>
<br>
"should there be no slave that is synchronised with the master,<br>
messages that only the master held will be lost."<br>
<br>
Messages can get lost if you recycle nodes in the cluster faster than it<br>
takes for the queue contents to be replaced entirely with new messages.<br>
The solution is to wait for slaves to become synchronised before<br>
restarting each node.<br>
<br>
The next version of RabbitMQ will feature manual eager synchronisation<br>
which will allow slaves to catch up on old messages held only on the<br>
master node. You can obtain this feature now by installing a nightly<br>
build of the broker, or wait for the next release.<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
-Emile<br>
<br>
<br>
<br>
<br>
<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>Jason McIntosh<br><a href="http://mcintosh.poetshome.com/blog/">http://mcintosh.poetshome.com/blog/</a><br>573-424-7612
</div>