<div dir="ltr">Simon,<div><br></div><div>Based on this:</div><div><br></div><div>> <span style="font-family:arial,sans-serif;font-size:12.727272033691406px">he key factor seems to be (at a TCP level) that A realises it is disconnected from B but not vice versa.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:12.727272033691406px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:12.727272033691406px">Should we expect a 3.2.5 at some point soon? I ask only because we're about to upgrade to 3.2.4 in production, but will hold off if there's yet more autoheal goodness coming.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:12.727272033691406px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:12.727272033691406px">Matt</span></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Tue, Mar 18, 2014 at 2:54 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi. Just to let you know that this has not been forgotten about. I have reproduced the problem you describe and am investigating it. The key factor seems to be (at a TCP level) that A realises it is disconnected from B but not vice versa. Therefore (at an Erlang level) the partition is not detected.<br>
<br>
It's not specifically to do with pause_minority mode.<br>
<br>
I'll post more information when I have it.<br>
<br>
Cheers, Simon<div><div class="h5"><br>
<br>
On 05/03/2014 17:18, David Rodrigues wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Dear RabbitMQ Community,<br>
<br>
I'm having some problems with the {cluster_partition_handling,<br>
pause_minority} option and I would like to share with you my doubts.<br>
<br>
First, the architecture. I'm running RabbitMQ 3.2.4 (Erlang R14A) on 3<br>
nodes (rabbitmq@rabbitmq-test01, rabbitmq@rabbitmq-test02 and<br>
rabbitmq@rabbitmq-test03) on a virtualized platform - quite similar to<br>
EC2. And because of that I have, from time to time, connectivity issues<br>
– I know, that's bad ;)<br>
<br>
Digging into the doc I have found that the best way to handle the<br>
problem is using the pause_minority option -<br>
<a href="https://www.rabbitmq.com/partitions.html" target="_blank">https://www.rabbitmq.com/<u></u>partitions.html</a>.<br>
<br>
But from time to time my nodes get disconnected from each other and do<br>
not recover automatically. Hopefully I have managed to reproduce the<br>
problem. Here are the steps.<br>
<br>
THE CONFIGURATION FILE<br>
******************************<u></u>*****************************<br>
<br>
My configuration file is quite simple:<br>
<br>
%% -*- mode: erlang -*-<br>
[<br>
{rabbit,<br>
[<br>
{auth_mechanisms, ['PLAIN', 'AMQPLAIN']},<br>
{default_vhost, <<"/">>},<br>
{default_user, <<"admin">>},<br>
{default_pass, <<"admin">>},<br>
{default_permissions, [<<".*">>, <<".*">>, <<".*">>]},<br>
{default_user_tags, [administrator]},<br>
{cluster_partition_handling, pause_minority},<br>
{cluster_nodes, {['rabbitmq@rabbitmq-test01',<br>
'rabbitmq@rabbitmq-test02', 'rabbitmq@rabbitmq-test03'], disc}}<br>
]},<br>
{kernel, []},<br>
{rabbitmq_management, []},<br>
{rabbitmq_management_agent, []},<br>
{rabbitmq_shovel,<br>
[{shovels, []}<br>
]},<br>
{rabbitmq_stomp, []},<br>
{rabbitmq_mqtt, []},<br>
{rabbitmq_amqp1_0, []},<br>
{rabbitmq_auth_backend_ldap, []}<br>
].<br>
<br>
Has you can see, the {cluster_partition_handling, pause_minority} option<br>
is there.<br>
<br>
PAUSE_MINORITY WORKING<br>
******************************<u></u>*****************************<br>
<br>
When the network outage is long enough, the option works perfectly.<br>
<br>
To simulate a connection problem on rabbitmq-test03 I run:<br>
<br>
iptables -A INPUT -s rabbitmq-test01 -j DROP; iptables -A OUTPUT -d<br>
rabbitmq-test01 -j DROP<br>
iptables -A INPUT -s rabbitmq-test02 -j DROP; iptables -A OUTPUT -d<br>
rabbitmq-test02 -j DROP<br>
<br>
Then wait long enough for the following messages to appear in the logs<br>
of rabbitmq-test03 (approximately 180 seconds):<br>
<br>
=ERROR REPORT==== 5-Mar-2014::16:51:02 ===<br>
** Node 'rabbitmq@rabbitmq-test02' not responding **<br>
** Removing (timedout) connection **<br>
=ERROR REPORT==== 5-Mar-2014::16:51:02 ===<br>
** Node 'rabbitmq@rabbitmq-test01' not responding **<br>
** Removing (timedout) connection **<br>
=INFO REPORT==== 5-Mar-2014::16:51:02 ===<br>
rabbit on node 'rabbitmq@rabbitmq-test02' down<br>
=WARNING REPORT==== 5-Mar-2014::16:51:30 ===<br>
Cluster minority status detected - awaiting recovery<br>
=INFO REPORT==== 5-Mar-2014::16:51:30 ===<br>
rabbit on node 'rabbitmq@rabbitmq-test01' down<br>
=INFO REPORT==== 5-Mar-2014::16:51:30 ===<br>
Stopping RabbitMQ<br>
=INFO REPORT==== 5-Mar-2014::16:51:30 ===<br>
stopped TCP Listener on [::]:5672<br>
=WARNING REPORT==== 5-Mar-2014::16:51:58 ===<br>
Cluster minority status detected - awaiting recovery<br>
<br>
When flushing the rules (iptables -F) the connectivity is reestablished<br>
and the cluster works perfectly.<br>
<br>
In the logs:<br>
<br>
=INFO REPORT==== 5-Mar-2014::16:52:58 ===<br>
started TCP Listener on [::]:5672<br>
=INFO REPORT==== 5-Mar-2014::16:52:58 ===<br>
rabbit on node 'rabbitmq@rabbitmq-test01' up<br>
=INFO REPORT==== 5-Mar-2014::16:52:58 ===<br>
rabbit on node 'rabbitmq@rabbitmq-test02' up<br>
<br>
Finally the cluster status :<br>
<br>
Cluster status of node 'rabbitmq@rabbitmq-test03' ...<br>
[{nodes,[{disc,['rabbitmq@<u></u>rabbitmq-test01','rabbitmq@<u></u>rabbitmq-test02',<br>
'rabbitmq@rabbitmq-test03']}]}<u></u>,<br>
{running_nodes,['rabbitmq@<u></u>rabbitmq-test01','rabbitmq@<u></u>rabbitmq-test02',<br>
'rabbitmq@rabbitmq-test03']},<br>
{partitions,[]}]<br>
...done.<br>
<br>
So far, so good. The option works flawlessly.<br>
<br>
PAUSE_MINORITY NOT WORKING<br>
******************************<u></u>*****************************<br>
<br>
Life is not so bright when the network partition is not long enough.<br>
<br>
On rabbitmq-test03 I will run my iptables commands again:<br>
<br>
iptables -A INPUT -s rabbitmq-test01 -j DROP; iptables -A OUTPUT -d<br>
rabbitmq-test01 -j DROP<br>
iptables -A INPUT -s rabbitmq-test02 -j DROP; iptables -A OUTPUT -d<br>
rabbitmq-test02 -j DROP<br>
<br>
However this time I'll only wait 60 seconds before flushing my rules<br>
with iptables -F.<br>
<br>
And that's the result in rabbitmq-test03 logs:<br>
<br>
=INFO REPORT==== 5-Mar-2014::16:55:00 ===<br>
rabbit on node 'rabbitmq@rabbitmq-test02' down<br>
=ERROR REPORT==== 5-Mar-2014::16:55:00 ===<br>
Mnesia('rabbitmq@rabbitmq-<u></u>test03'): ** ERROR ** mnesia_event got<br>
{inconsistent_database, running_partitioned_network,<br>
'rabbitmq@rabbitmq-test02'}<br>
=INFO REPORT==== 5-Mar-2014::16:55:00 ===<br>
rabbit on node 'rabbitmq@rabbitmq-test01' down<br>
=INFO REPORT==== 5-Mar-2014::16:55:00 ===<br>
Statistics database started.<br>
=INFO REPORT==== 5-Mar-2014::16:55:00 ===<br>
Statistics database started.<br>
<br>
Again, the result is quiet ugly on rabbitmq-test01:<br>
<br>
=ERROR REPORT==== 5-Mar-2014::16:55:00 ===<br>
** Node 'rabbitmq@rabbitmq-test03' not responding **<br>
** Removing (timedout) connection **<br>
=INFO REPORT==== 5-Mar-2014::16:55:00 ===<br>
rabbit on node 'rabbitmq@rabbitmq-test03' down<br>
=INFO REPORT==== 5-Mar-2014::16:55:01 ===<br>
global: Name conflict terminating {rabbit_mgmt_db,<2669.1582.0>}<br>
<br>
Finally my cluster status:<br>
<br>
Cluster status of node 'rabbitmq@rabbitmq-test03' ...<br>
[{nodes,[{disc,['rabbitmq@<u></u>rabbitmq-test01','rabbitmq@<u></u>rabbitmq-test02',<br>
'rabbitmq@rabbitmq-test03']}]}<u></u>,<br>
{running_nodes,['rabbitmq@<u></u>rabbitmq-test03']},<br>
{partitions,[{'rabbitmq@<u></u>rabbitmq-test03',['rabbitmq@<u></u>rabbitmq-test02']}]}]<br>
...done.<br>
<br>
That's it. Even with the pause_minority option my cluster was disintegrated.<br>
<br>
SYNOPSIS<br>
******************************<u></u>*****************************<br>
<br>
In short, if the network outage is long enough everything goes according<br>
to the plan and the cluster works perfectly once the connectivity is<br>
reestablished. However if the network outage has a intermediate duration<br>
(not too short, not too long) the pause_minority option seems not to work.<br>
<br>
Are you aware of this problem? Is there any solution to cope with this<br>
particular situation?<br>
<br>
Thanks,<br>
David<br>
<br>
<br></div></div>
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
<br>
</blockquote>
<br>
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
</blockquote></div><br></div>