<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Nov 25, 2013 at 9:39 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Graeme: we recently merged a few fixes to bugs that we found as a result of running your test scripts; these can be picked up in the latest nightlies. I am now able to run rebalance_cluster.sh and toggle_nodes.sh indefinitely without anything breaking.<br>
</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks for producing these scripts!<br></blockquote><br></div><div class="gmail_quote">No problem!<br><br>However, when I retested with the latest nightly build (3.2.1.41125-1), I ran into issues. I used my scripts to setup a 5 node local cluster, made 100 queues, and populated them with 1000 1MB messages each. Then ran the rebalance script, and every node promptly crashed once it started applying the balancing policies. When I brought them back up, the cluster thrashed the disks for a bit applying the policies, but when it finished, only 53,949 / 100,000 messages were actually present.<br>
<br>I suspected this was because I wasn't using confirmed message delivery in my Go queue populator, so I changed it to do message confirmations and re-tried. This didn't result in a crash and all the policies get created, but none of them auto apply, even though they're all set to "automatic". The cluster just sits there, and never actually applies any of the polices or distributes any data, even if I run through every queue and run sync_queue.<br>
<br>So it appears there's still bugs with actually applying policies and syncing data, as I actually can't get any policies to properly replicate without using unconfirmed messages and crashing the cluster. I've uploaded the current version of the scripts I'm using to Mega: <br>
</div><div class="gmail_quote"><br><a href="https://mega.co.nz/#!jsBH2JSR!fP1zVwsi1ScQrnwzNOBlH05HJN0z9IofYRzy4WvjBjU">https://mega.co.nz/#!jsBH2JSR!fP1zVwsi1ScQrnwzNOBlH05HJN0z9IofYRzy4WvjBjU</a><br><br></div><div class="gmail_quote">
The steps I used to create the situation where there are policies that refuse to apply or sync should be easy to replicate by doing the following:<br><br></div><div class="gmail_quote">./create_cluster.sh<br></div><div class="gmail_quote">
./setup_queues.sh<br></div><div class="gmail_quote">./populate_queues.sh<br></div><div class="gmail_quote">./rebalance_cluster.sh<br><br></div><div class="gmail_quote">Graeme<br><br></div></div></div>