Alex, I believe what you are seeing is because mirrored queues do not automatically sync.<div><ul><li>When you restart node1 it has 0 messages and is not synced with node2. Since node1 was offline, the durable messages in it's queue are considered out dated, since node2 is more likely to be up to date, and are discarded. Node1 is now a slave of node2, but the messages in node2 are not copied to node1.<br></li><li>Then when you restart node2, node1 with 0 messages becomes the master.<br></li></ul></div><div>This can feel odd, because often you bounce one box and then the other for maintenance or similar. You need to make sure that all messages in the queue when one node is bounced have been drained before bouncing the other node. You can verify this in a few ways, but the easiest is to just peek at the Admin console and your queue will show -1 (or -n for n nodes) rather than +1. This tells you that the mirrors are not synced.</div><div><br><br>On Wednesday, January 9, 2013 1:57:54 PM UTC-5, Alexandre Bunn wrote:<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;">Good afternoon<br><br>I have a 2 nodes cluster with mirrored queues and I'm setting delivery_mode = 2 when I'm publishing messages to the cluster. I'm missing messages when I follow these steps below (for test issues).<br><br>Step 0 - Node1 (master) Node2 (slave)<br><br># rabbitmqctl cluster_status<br>Cluster status of node 'rabbit@vid-mq01-mia' ...<br>[{nodes,[{disc,['rabbit@vid-<wbr>mq01-mia','rabbit@vid-mq02-mia<wbr>']}]},<br> {running_nodes,['rabbit@vid-<wbr>mq02-mia','rabbit@vid-mq01-mia<wbr>']},<br> {partitions,[]}]<br>...done.<br><br># rabbitmqctl list_policies<br>Listing policies ...<br>/ ha-all ^ha\\. {"ha-mode":"all"} 0<br>...done.<br><br>## rabbitmqctl list_queues name durable policy<br>Listing queues ...<br>ha.teste true ha-all<br>...done.<br><br><br>- Step 1 - I write 2 messages, at ha.teste queue, 1 on each member of the cluster and the messages are mirrored as expected. Here there are two messages on each nodes (rabbitmqctl list_queues)<br><br>-- node1<br># rabbitmqctl list_queues<br>Listing queues ...<br>ha.teste 2<br>...done.<br># rabbitmqctl list_queues name pid slave_pids synchronised_slave_pids<br>Listing queues ...<br>ha.teste <'rabbit@vid-mq01-mia'.3.253.<wbr>0> [<'rabbit@vid-mq02-mia'.2.252.<wbr>0>] [<'rabbit@vid-mq02-mia'.2.252.<wbr>0>]<br>...done.<br><br>-- node2<br># rabbitmqctl list_queues<br>Listing queues ...<br>ha.teste 2<br>...done.<br># rabbitmqctl list_queues name pid slave_pids synchronised_slave_pids<br>Listing queues ...<br>ha.teste <'rabbit@vid-mq01-mia'.3.253.<wbr>0> [<'rabbit@vid-mq02-mia'.2.252.<wbr>0>] [<'rabbit@vid-mq02-mia'.2.252.<wbr>0>]<br>...done.<br><br>- Step 2 - I've stopped rabbitmq-server on node1 and node2 assume the master. Here there are the two messages on node2<br><br>-- Logs on node2<br>Mirrored-queue (queue 'ha.teste' in vhost '/'): Promoting slave <'rabbit@vid-mq02-mia'.2.252.<wbr>0> to master<br><br>-- Status of the cluster on node2<br><br># rabbitmqctl cluster_status<br>Cluster status of node 'rabbit@vid-mq02-mia' ...<br>[{nodes,[{disc,['rabbit@vid-<wbr>mq01-mia','rabbit@vid-mq02-mia<wbr>']}]},<br> {running_nodes,['rabbit@vid-<wbr>mq02-mia']},<br> {partitions,[]}]<br>...done.<br><br>-- Status of the ha.teste queue on node2<br><br># rabbitmqctl list_queues<br>Listing queues ...<br>ha.teste 2<br>...done.<br># rabbitmqctl list_queues name pid slave_pids synchronised_slave_pids<br>Listing queues ...<br>ha.teste <'rabbit@vid-mq02-mia'.2.252.<wbr>0> [] []<br>...done.<br><br>- Step 3 - I start the rabbitmq-server on node1, node2 still master and node1 is running as slave. Here there are the two messages on node1 and node2<br><br>-- Log on node2<br>=INFO REPORT==== 9-Jan-2013::18:40:46 ===<br>rabbit on node 'rabbit@vid-mq01-mia' up<br><br>-- Status of the cluster on node1<br><br># rabbitmqctl cluster_status<br>Cluster status of node 'rabbit@vid-mq01-mia' ...<br>[{nodes,[{disc,['rabbit@vid-<wbr>mq01-mia','rabbit@vid-mq02-mia<wbr>']}]},<br> {running_nodes,['rabbit@vid-<wbr>mq02-mia','rabbit@vid-mq01-mia<wbr>']},<br> {partitions,[]}]<br>...done.<br><br>-- Status of the ha.teste queue on node2<br># rabbitmqctl list_queues<br>Listing queues ...<br>ha.teste 2<br>...done.<br><br># rabbitmqctl list_queues name pid slave_pids synchronised_slave_pids<br>Listing queues ...<br>ha.teste <'rabbit@vid-mq02-mia'.2.252.<wbr>0> [<'rabbit@vid-mq01-mia'.1.253.<wbr>0>] []<br>...done.<br><br>- Step 4 - I've stopped rabbitmq-server on node2 and node1 assume the master but the messages disapeared<br><br>-- Logs on node1<br><br>=INFO REPORT==== 9-Jan-2013::18:43:42 ===<br>Mirrored-queue (queue 'ha.teste' in vhost '/'): Promoting slave <'rabbit@vid-mq01-mia'.1.253.<wbr>0> to master<br><br>=INFO REPORT==== 9-Jan-2013::18:43:42 ===<br>rabbit on node 'rabbit@vid-mq02-mia' down<br><br>-- Cluster status on node1<br><br># rabbitmqctl cluster_status<br>Cluster status of node 'rabbit@vid-mq01-mia' ...<br>[{nodes,[{disc,['rabbit@vid-<wbr>mq01-mia','rabbit@vid-mq02-mia<wbr>']}]},<br> {running_nodes,['rabbit@vid-<wbr>mq01-mia']},<br> {partitions,[]}]<br>...done.<br><br>-- Status of the ha.teste queue on node1<br><br># rabbitmqctl list_queues<br>Listing queues ...<br>ha.teste 0<br>...done.<br><br># rabbitmqctl list_queues name pid slave_pids synchronised_slave_pids<br>Listing queues ...<br>ha.teste <'rabbit@vid-mq01-mia'.1.253.<wbr>0> [] []<br>...done.<br><br> Is it expected? What we have to do at server side or client side
(python script) to make the messages really durable? All the servers are
running centos6 x86_64 and rabbitmq 3.0.1-1.<br><br>Thanks<br><br>Alexandre Bunn<br></blockquote></div>