[rabbitmq-discuss] active/active message persistence problem

Tue Nov 22 06:54:57 GMT 2011

You can run "rabbitmqctl list_queues name slave_pids
synchronised_slave_pids" to see if the cluster is synchronized or not
but I can tell you right now that they will not be. In step 3 when you
start vm1 back up and it is the slave. It is telling you that there
are 10 messages in the vm2 queue. When you talk to a broker in a
cluster it will talk to the master queue. VM1 will not be synchronized
until all 10 messages are read out of the vm2 queue, because rabbitmq
mirrored clusters do not read old messages that are already in the
master queue. The slave reads the tail of the new message being sent
to the master and expects that once it has been long enough then it
will catch up to the same state as the master.

http://www.rabbitmq.com/ha.html "Unsynchronised Slaves" I think does a
good job of explaining it.

Cameron

On Tue, Nov 22, 2011 at 12:37 AM, Jeffrey Chen <cpthk at hotmail.com> wrote:
> Hello Cameron:
>
> To resolve any confusion, let me make a table to clarify. I DO have
> "x-ha-policy" set to "all"
>
>   Steps    |   vm1                                       |        vm2
>
> ------------------------------------------------------------------------------
>   1.         |  started (10 msg, disk, master)   |  started (10 msg, disk)
>   2.         |  stop                                         |  started (10
> msg, disk, became master)
>   3.         |  started (10 msg, disk, slave)      |  started (10 msg, disk,
> master)
>   4.         |  started (0 msg, disk, master)     | stop
>
> Noticed that in my step 4, when I stop my VM2, the message in VM1
> immediately disappear. You are right, each step has interval of about 10
> seconds, I didn't wait very long to stop the new master. You mentioned that
> I have to I have to wait until the queues are synchronized. How long is that
> waiting? Is there a command that I could find out if they finished the
> synchronization? Like rabbitmqctl command or anything? Or any way to force
> synchronize right now. I am only testing 10 messages, it shouldn't take that
> long.
>
> Thanks for your help.
>
> Jeffrey.
>
>> From: cameron.davison at gmail.com
>> Date: Mon, 21 Nov 2011 23:46:24 -0600
>> Subject: Re: [rabbitmq-discuss] active/active message persistence problem
>> To: cpthk at hotmail.com
>> CC: rabbitmq-discuss at lists.rabbitmq.com
>>
>> What order are you restarting the nodes? Any rabbitmq server that
>> connects to a cluster will start its queue fresh and begin replicating
>> from the master. You should wait and make sure that all the queues are
>> synchronized before you shut down the master, or new master. Old
>> messages are not read from nodes that connect to the cluster because
>> the assumption is that the queue will go to 0 soon and then the old
>> messages will not matter so only the new messages will be important.
>> Also, you should make sure that the "x-ha-policy" is set to "all" or
>> to the "nodes" that you would like the queue to be mirrored to.
>>
>> Cameron
>>
>> On Mon, Nov 21, 2011 at 7:14 PM, Jeffrey Chen <cpthk at hotmail.com> wrote:
>> > Hi RabbitMQ community:
>> >
>> > I am encounter this problem about message persistence. Here is my case,
>> > I
>> > have 2 server nodes with active/active setup, both are persisted to
>> > DISK.
>> > The queue declarations are defined as durable.
>> > This is what I got when doing:
>> >
>> > # rabbitmqctl cluster_status
>> > Cluster status of node 'rabbit at aaron-vm177' ...
>> > [{nodes,[{disc,['rabbit at vm1','rabbit at vm2']}]},
>> >  {running_nodes,['rabbit at vm1','rabbit at vm2']}]
>> > ...done.
>> >
>> > # rabbitmqctl list_queues
>> > Listing queues ...
>> > test    10
>> > ...done.
>> >
>> >
>> > My vm1 is currently master node. When I stop RabbitMQ service in vm1,
>> > and
>> > start it back up. I check my messages in the queue, the messages are
>> > still
>> > there. This time when I stop RabbitMQ service in vm2. The messages will
>> > be
>> > gone. It seems like when I restart both server nodes, the data will be
>> > lost.
>> > Does anyone know the reason? Is there any way to work around this?
>> >
>> > My goal is that any server nodes down, and when I bring it back up, no
>> > data
>> > will be lost.
>> >
>> > Jeffrey.
>> >
>> > _______________________________________________
>> > rabbitmq-discuss mailing list
>> > rabbitmq-discuss at lists.rabbitmq.com
>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >
>> >
>