[rabbitmq-discuss] problem with an HA pair of rabbitmq servers

[allan bailey]

We have a pair of rabbitmq servers.   The 1st server periodically does a lot
of intense I/O copying data
out to the 2nd server.   This apparently causes timeouts that then cause a
partitioning of the cluster.

My main question is this:  can we set the timeout value higher?  and if so,
how?  I found nothing in the
manual pages about a timeout setting between cluster nodes.

When we see this problem, the quickest solution is to shutdown the 2nd
server, nuke the mnesia dir and
rebuild it w/ a cluster file pointing to the 1st server.  Then start up
again and all is well.  (it even gets the users and vhosts from the 1st.)


Below are the errors we see when this problem happens.


1st server sees this error:
===
[=ERROR REPORT==== 28-Feb-2010::16:28:52 ===
Mnesia(rabbit at rq101): ** ERROR ** mnesia_event got {inconsistent_database,
running_partitioned_network, rabbit at rq102}]

===


2nd server sees these errors:
===
[=ERROR REPORT==== 28-Feb-2010::16:27:08 ===
** Node rabbit at rq101 not responding **
** Removing (timedout) connection **]
[=ERROR REPORT==== 28-Feb-2010::16:28:38 ===
** Node rabbit at rq101 not responding **
** Removing (timedout) connection **]
=INFO REPORT==== 28-Feb-2010::16:28:52 ===
node rabbit at rq101 up
=WARNING REPORT==== 28-Feb-2010::16:28:52 ===
The global_name_server locker process received an unexpected message:
{{#Ref<0.0.0.186122>,rabbit at rq101},true}
=WARNING REPORT==== 28-Feb-2010::16:28:52 ===
The global_name_server locker process received an unexpected message:
{{#Ref<0.0.0.186227>,rabbit at rq101},true}
[=ERROR REPORT==== 28-Feb-2010::16:28:52 ===
Mnesia(rabbit at rq102): ** ERROR ** mnesia_event got {inconsistent_database,
running_partitioned_network, rabbit at rq101}]

===


thanks,
Allan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100228/0b7d5746/attachment.htm