[rabbitmq-discuss] Mirrored queue failover

[Matthew Sackman]

Hi,

On Thu, Mar 29, 2012 at 05:36:07PM +0900, Katsushi Fukui wrote:
> I'm a RabbitMQ newbie and am trying to use mirrored queue in the
> cluster (3 disc nodes on the 3 servers). I use RabbitMQ 2.8.1, Erlang
> R15B, CentOS 6.2. I wanted to see the behavior of the failover in the
> case of any issue, so I tested some scenarios and I have a little
> confusion with the case below:
> 
> a) 3 nodes (rabbit1, rabbit2, rabbit3) are running, and I created que1
> on the rabbit1 with an argument "x-ha-policy":"all". So now que1 have
> a master node "rabbit1 and slave nodes "rabbit2" and "rabbit3", I
> confirmed it with the command "rabbitctl list_queues name durable pid
> slave_pids synchronised_slave_pids".
> 
> b) I stopped one of the slave node "rabbit2" with "rabbitctl stop".
> Now que1 has only one slave node "rabbit3".
> 
> c) Next, I started the node "rabbit2" using "rabbit-server -detached"
> and checked the cluster status with "rabbitctl cluster_status". It
> seems good 3 nodes has 3 disc nodes and running. But when I checked
> the queue status, que1 has just one slave "rabbit3". I restarted
> rabbit2 again, but rabbit2 doesn't become a slave of que1, also I can
> find a log like this:
> =ERROR REPORT==== 29-Mar-2012::15:12:40 ===
> Discarding message {'$gen_call',{<0.1876.0>,#Ref<0.0.0.8584>},info}
> from <0.1876.0> to <0.426.0> in an old incarnation (2) of this node
> (3)
> 
> In my understanding, when a new node is added in the cluster, that
> node will become a slave of the mirrored queues in that cluster. Is
> this correct?

Your understanding is correct. I am not sure why this problem has
occurred. The Error Report above is harmless, and is really just
documenting Erlang sorting itself out. What is the result of

rabbitmqctl list_queues name pid slave_pids

?

Also, if that suggests there are not 2 slaves, then could you check
through and send us (probably off-list) the logs of both rabbit2 as it
restarts, and for the same time period from rabbit1 and 3?

Matthew