In general your queues should not be "stopped" so this would not be a problem. In practice there can be reasons this occurs. I would look first at minimizing the amount of time a message spends in a queue. If you have error queues that hold messages, don't use rabbit to store these. Read them off and store them in a db.<div>If you absolutely feel you must be able to sync queues, the best I know of is to create a process that reads and requeues the message. You are likely to lose message order if you have new messages coming in.<br><br>On Friday, January 11, 2013 5:51:38 AM UTC-5, Alexandre Bunn wrote:<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;">I've found in a forum (<a href="http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/12438" target="_blank">http://comments.gmane.org/<wbr>gmane.comp.networking.<wbr>rabbitmq.general/12438</a>) a user that have reported the same situation and an user have answered:<br><br>"<br><pre>You can run "rabbitmqctl list_queues name slave_pids
synchronised_slave_pids" to see if the cluster is synchronized or not
but I can tell you right now that they will not be. In step 3 when you
start vm1 back up and it is the slave. It is telling you that there
are 10 messages in the vm2 queue. When you talk to a broker in a
cluster it will talk to the master queue. VM1 will not be synchronized
until all 10 messages are read out of the vm2 queue, because rabbitmq
mirrored clusters do not read old messages that are already in the
master queue. The slave reads the tail of the new message being sent
to the master and expects that once it has been long enough then it
will catch up to the same state as the master.
<a rel="nofollow" href="http://www.rabbitmq.com/ha.html" target="_blank">http://www.rabbitmq.com/ha.<wbr>html</a> "Unsynchronised Slaves" I think does a
good job of explaining it.<br>"<br></pre><br>Does rabbitmq continues with this behavior? Is there any configuration that I can do to solve this issue?<br><br>Thanks<br><br>Alexandre<br><br>Em quinta-feira, 10 de janeiro de 2013 21h29min24s UTC-2, Scott Brown escreveu:<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex">Alex, I believe what you are seeing is because mirrored queues do not automatically sync.<div><ul><li>When you restart node1 it has 0 messages and is not synced with node2. Since node1 was offline, the durable messages in it's queue are considered out dated, since node2 is more likely to be up to date, and are discarded. Node1 is now a slave of node2, but the messages in node2 are not copied to node1.<br></li><li>Then when you restart node2, node1 with 0 messages becomes the master.<br></li></ul></div><div>This can feel odd, because often you bounce one box and then the other for maintenance or similar. You need to make sure that all messages in the queue when one node is bounced have been drained before bouncing the other node. You can verify this in a few ways, but the easiest is to just peek at the Admin console and your queue will show -1 (or -n for n nodes) rather than +1. This tells you that the mirrors are not synced.</div><div><br><br>On Wednesday, January 9, 2013 1:57:54 PM UTC-5, Alexandre Bunn wrote:<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex">Good afternoon<br><br>I have a 2 nodes cluster with mirrored queues and I'm setting delivery_mode = 2 when I'm publishing messages to the cluster. I'm missing messages when I follow these steps below (for test issues).<br><br>Step 0 - Node1 (master) Node2 (slave)<br><br># rabbitmqctl cluster_status<br>Cluster status of node 'rabbit@vid-mq01-mia' ...<br>[{nodes,[{disc,['rabbit@vid-<wbr>mq01-mia','rabbit@vid-mq02-mia<wbr>']}]},<br> {running_nodes,['rabbit@vid-<wbr>mq02-mia','rabbit@vid-mq01-mia<wbr>']},<br> {partitions,[]}]<br>...done.<br><br># rabbitmqctl list_policies<br>Listing policies ...<br>/ ha-all ^ha\\. {"ha-mode":"all"} 0<br>...done.<br><br>## rabbitmqctl list_queues name durable policy<br>Listing queues ...<br>ha.teste true ha-all<br>...done.<br><br><br>- Step 1 - I write 2 messages, at ha.teste queue, 1 on each member of the cluster and the messages are mirrored as expected. Here there are two messages on each nodes (rabbitmqctl list_queues)<br><br>-- node1<br># rabbitmqctl list_queues<br>Listing queues ...<br>ha.teste 2<br>...done.<br># rabbitmqctl list_queues name pid slave_pids synchronised_slave_pids<br>Listing queues ...<br>ha.teste <'rabbit@vid-mq01-mia'.3.253.<wbr>0> [<'rabbit@vid-mq02-mia'.2.252.<wbr>0>] [<'rabbit@vid-mq02-mia'.2.252.<wbr>0>]<br>...done.<br><br>-- node2<br># rabbitmqctl list_queues<br>Listing queues ...<br>ha.teste 2<br>...done.<br># rabbitmqctl list_queues name pid slave_pids synchronised_slave_pids<br>Listing queues ...<br>ha.teste <'rabbit@vid-mq01-mia'.3.253.<wbr>0> [<'rabbit@vid-mq02-mia'.2.252.<wbr>0>] [<'rabbit@vid-mq02-mia'.2.252.<wbr>0>]<br>...done.<br><br>- Step 2 - I've stopped rabbitmq-server on node1 and node2 assume the master. Here there are the two messages on node2<br><br>-- Logs on node2<br>Mirrored-queue (queue 'ha.teste' in vhost '/'): Promoting slave <'rabbit@vid-mq02-mia'.2.252.<wbr>0> to master<br><br>-- Status of the cluster on node2<br><br># rabbitmqctl cluster_status<br>Cluster status of node 'rabbit@vid-mq02-mia' ...<br>[{nodes,[{disc,['rabbit@vid-<wbr>mq01-mia','rabbit@vid-mq02-mia<wbr>']}]},<br> {running_nodes,['rabbit@vid-<wbr>mq02-mia']},<br> {partitions,[]}]<br>...done.<br><br>-- Status of the ha.teste queue on node2<br><br># rabbitmqctl list_queues<br>Listing queues ...<br>ha.teste 2<br>...done.<br># rabbitmqctl list_queues name pid slave_pids synchronised_slave_pids<br>Listing queues ...<br>ha.teste <'rabbit@vid-mq02-mia'.2.252.<wbr>0> [] []<br>...done.<br><br>- Step 3 - I start the rabbitmq-server on node1, node2 still master and node1 is running as slave. Here there are the two messages on node1 and node2<br><br>-- Log on node2<br>=INFO REPORT==== 9-Jan-2013::18:40:46 ===<br>rabbit on node 'rabbit@vid-mq01-mia' up<br><br>-- Status of the cluster on node1<br><br># rabbitmqctl cluster_status<br>Cluster status of node 'rabbit@vid-mq01-mia' ...<br>[{nodes,[{disc,['rabbit@vid-<wbr>mq01-mia','rabbit@vid-mq02-mia<wbr>']}]},<br> {running_nodes,['rabbit@vid-<wbr>mq02-mia','rabbit@vid-mq01-mia<wbr>']},<br> {partitions,[]}]<br>...done.<br><br>-- Status of the ha.teste queue on node2<br># rabbitmqctl list_queues<br>Listing queues ...<br>ha.teste 2<br>...done.<br><br># rabbitmqctl list_queues name pid slave_pids synchronised_slave_pids<br>Listing queues ...<br>ha.teste <'rabbit@vid-mq02-mia'.2.252.<wbr>0> [<'rabbit@vid-mq01-mia'.1.253.<wbr>0>] []<br>...done.<br><br>- Step 4 - I've stopped rabbitmq-server on node2 and node1 assume the master but the messages disapeared<br><br>-- Logs on node1<br><br>=INFO REPORT==== 9-Jan-2013::18:43:42 ===<br>Mirrored-queue (queue 'ha.teste' in vhost '/'): Promoting slave <'rabbit@vid-mq01-mia'.1.253.<wbr>0> to master<br><br>=INFO REPORT==== 9-Jan-2013::18:43:42 ===<br>rabbit on node 'rabbit@vid-mq02-mia' down<br><br>-- Cluster status on node1<br><br># rabbitmqctl cluster_status<br>Cluster status of node 'rabbit@vid-mq01-mia' ...<br>[{nodes,[{disc,['rabbit@vid-<wbr>mq01-mia','rabbit@vid-mq02-mia<wbr>']}]},<br> {running_nodes,['rabbit@vid-<wbr>mq01-mia']},<br> {partitions,[]}]<br>...done.<br><br>-- Status of the ha.teste queue on node1<br><br># rabbitmqctl list_queues<br>Listing queues ...<br>ha.teste 0<br>...done.<br><br># rabbitmqctl list_queues name pid slave_pids synchronised_slave_pids<br>Listing queues ...<br>ha.teste <'rabbit@vid-mq01-mia'.1.253.<wbr>0> [] []<br>...done.<br><br> Is it expected? What we have to do at server side or client side
(python script) to make the messages really durable? All the servers are
running centos6 x86_64 and rabbitmq 3.0.1-1.<br><br>Thanks<br><br>Alexandre Bunn<br></blockquote></div></blockquote></blockquote></div>