[rabbitmq-discuss] Queue disappear from nodes in clusters

[Simon MacMullen]

On 26/07/2013 12:54AM, vantheshark wrote:
> Hi Simon,
>
> I think the log files are too big (2Gb) to upload here, even after I
> compress it.

Hmm; what about the -sasl.log files? Those should contain the crashes in 
question.

> However, these are some lines like this in node2
> /
> Mirrored-queue (queue 'MyQueue_Priority3' in vhost '/'): Master
> <rabbit at queue2.2.2114.1> saw deaths of mirrors <rabbit at queue3.1.365.1>
> /
>
> "MyQueue_Priority3" was the one vanished.

That's just a normal message from queue mirroring - we log whenever a 
mirror dies.

> I assume all exchanges and queues are mirrored across nodes in
> this cluster so even 1 node down, the data should not be lost?

Exchanges are always mirrored. Queues are if you turn on queue mirroring 
(which it sounds like you have).

But there is evidence of something crashing in the logs you posted, so I 
want to see what's going on there. I suspect that's why the queue 
vanished. Certainly it shouldn't just do that.

> I also used wan IPs and put the IPs in the host file before clustering, is
> it the problem.

Well, be aware that clustering is in general not partition tolerant. If 
you've experienced network partitions then you can expect things not to 
work flawlessly. Not that I would expect to see queues crash, but in 
general the behaviour is less well defined when partitioned.

<snip>

> I may choose *pause_minority* then.

Sure, but make sure you understand what pause_minority does. None of the 
cluster_partition_handling options are magic, you will always experience 
problems when a partition occurs - you just get a choice of what sort of 
problems.

> I will change to use the private IPs
> (provided by the host provider), hopefully will be better.

Private IPs are unlikely to make a difference.

> By the way, do
> you recommend a clustering of 3 nodes or 2 nodes?

Depends what you're trying to do. If you want to use pause_minority mode 
then you need at least three nodes.

Cheers, Simon