<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>I don't suppose you can post the code that you're using to trigger this can you?</div><div><br></div><div>Cheers,</div><div>Tim</div><br><div><div>On 27 Mar 2014, at 22:15, Matt Pietrek wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Because we're sometimes just mean to our software, I wrote a torture test to see how RabbitMQ's Autoheal deal with repeated partitions.<div><br></div><div>In a nutshell, we start with two brokers (3.2.4) in a cluster. I run my test which uses "iptables" to knock out the link between the two brokers and then restore things.</div>
<div><br></div><div>It does this break/fix continuously in a loop. The time between partitions, and the time inside partitions is configurable.</div><div><br></div><div>Using 60 seconds between inducing a partition, and 60 seconds in a partitioned state, I expect that this might be messy - The brokers try to autoheal, and then everything falls apart. However, I'd expect that once I stop my torture and return things back to "normal", that an autoheal will eventually succeed and the brokers will be happily clustered again.</div>
<div><br></div><div>This isn't what happens. Instead, the two brokers essentially ignore each other. Even after waiting for 10+ minutes. I can see each broker, but they each think the other is missing.</div><div><br></div>
<div>Here's a filtered view of the logs, grepping for "Autoheal|Starting|Stopping|Partitions|Winner|Loser":</div><div><br></div><div><p class=""><a href="mailto:rabbit@mq2.log">rabbit@mq2.log</a>: Autoheal request sent to rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq2.log">rabbit@mq2.log</a>: Autoheal: I am the winner, waiting for [rabbit@mq1] to stop</p><p class=""><a href="mailto:rabbit@mq2.log">rabbit@mq2.log</a>: Autoheal: I am the winner, waiting additionally for [rabbit@mq1] to stop</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal request sent to rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal request received from rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal decision</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Partitions: [[rabbit@mq1],[rabbit@mq2]]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Winner: rabbit@mq2</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Losers: [rabbit@mq1]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal request received from rabbit@mq2</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal decision</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Partitions: [[rabbit@mq1],[rabbit@mq2]]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Winner: rabbit@mq2</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Losers: [rabbit@mq1]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal: we were selected to restart; winner is rabbit@mq2</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Stopping RabbitMQ</p><p class=""><a href="mailto:rabbit@mq2.log">rabbit@mq2.log</a>: Autoheal: aborting - rabbit@mq1 went down</p><p class=""><a href="mailto:rabbit@mq2.log">rabbit@mq2.log</a>: Autoheal request sent to rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq2.log">rabbit@mq2.log</a>: Autoheal: we were selected to restart; winner is rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq2.log">rabbit@mq2.log</a>: Stopping RabbitMQ</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal: aborting - rabbit@mq2 went down</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal request sent to rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal request received from rabbit@mq2</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal decision</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Partitions: [[rabbit@mq1],[rabbit@mq2]]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Winner: rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Losers: [rabbit@mq2]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal request received from rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal decision</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Partitions: [[rabbit@mq1],[rabbit@mq2]]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Winner: rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Losers: [rabbit@mq2]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal: I am the winner, waiting for [rabbit@mq2] to stop</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal: I am the winner, waiting additionally for [rabbit@mq2] to stop</p><p class=""><a href="mailto:rabbit@mq2.log">rabbit@mq2.log</a>: Autoheal: aborting - rabbit@mq1 went down</p><p class=""><a href="mailto:rabbit@mq2.log">rabbit@mq2.log</a>: Autoheal request sent to rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq2.log">rabbit@mq2.log</a>: Autoheal: we were selected to restart; winner is rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal: aborting - rabbit@mq2 went down</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal request sent to rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal request received from rabbit@mq2</p><p class="">
<a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal decision</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Partitions: [[rabbit@mq1],[rabbit@mq2]]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Winner: rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Losers: [rabbit@mq2]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal request received from rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal decision</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Partitions: [[rabbit@mq1],[rabbit@mq2]]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Winner: rabbit@mq1</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: * Losers: [rabbit@mq2]</p><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal: I am the winner, waiting for [rabbit@mq2] to stop</p><div>
<br class="webkit-block-placeholder"></div><p class=""><a href="mailto:rabbit@mq1.log">rabbit@mq1.log</a>: Autoheal: I am the winner, waiting additionally for [rabbit@mq2] to stop</p><p class=""># And nothing else beyond this, even after waiting for 10+ minutes.</p><p class="">I don't ever see the "Stopping RabbitMQ" that I've seen in other Autoheal circumstances.</p><p class="">I can send more complete logs, but wanted to see if this is a known issue or expected behavior first.</p><p class=""><br></p><p class="">Matt</p></div></div>
_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss<br></blockquote></div><br></body></html>