[rabbitmq-discuss] Statistics database could not be contacted.

[Simon MacMullen]

On 29/10/12 17:36, Farshad Meshkinpour wrote:
> Hi Simon,
> We got 3 nodes in a cluster.  We upgraded one of data center switches on
> Wed night.  The rabbit log indicates loss of connectivity for a short
> time.  After that the node with stats says the other two nodes are down.
>   The other 2 nodes complaints that they couldn't access stats database!
>
> I tried to restart the node with stats database, but it wouldn't start &
> I had to disable management plugin in order to bring the node up.
> I tried to restart one of the other nodes and it wouldn't start unless
> disabling management plugin.
> Then after a day of trying various changed we enabled management-agent
> plugins and each node management came up.
>
> So, what is the recommended procedure and best practices in this situation?

Firstly, upgrade to 2.8.7 since there was a bug in mgmt database 
failover in earlier versions.

> Also, from time-to-time we have to maintain the switches & routers which
> may interrupt the connectivity.  Is there a best practice to plan ahead
> to avoid this type of issues?

RabbitMQ clustering does not tolerate network partitions well. If you 
are planning on causing a network partition, take down all the bits of 
the cluster that will be partitioned first.

Cheers, Simon

>
> Thanks,
> Farshad
>
>
>
>
>
>
> On Fri, Oct 26, 2012 at 9:18 AM, Simon MacMullen <simon at rabbitmq.com
> <mailto:simon at rabbitmq.com>> wrote:
>
>     On 25/10/12 20:03, Farshad Meshkinpour wrote:
>
>         I am running RMQ 2.8.6 in a three node cluster.  The first
>         rabbit node
>         that had statistics database went down.   I had to disable
>         management-plug to restart the node.
>
>         How do I get statistics database restarted?
>
>
>     It should restart automatically. However, there was a bug, fixed in
>     2.8.7, in which it could fail to fail over correctly when the node
>     it was originally running on went down. Is this what happened to
>     you? I'm not sure I fully understand your message.
>
>     Cheers, Simon
>
>     --
>     Simon MacMullen
>     RabbitMQ, VMware
>
>


-- 
Simon MacMullen
RabbitMQ, VMware