<div dir="ltr"><div><div><div>Ok, so here's the issues I'm still seeing with the latest nightly build (rabbitmq-server-3.2.0.41104-1.noarch):<br><br></div>- multiple simultaneous API calls on the CLI from running rabbitmqctl and amqp-delete-queue in parallel from bash still cause weird buggy responses from RMQ. running my setup_queues.sh multiple times is enough to trigger it. Seems to be primarily running the operations in the first loop that have problems, the second loop (which creates the queues) seems to run fine in parallel, which is good.<br>
</div><div>- this is also the case with my rebalance_cluster.sh script when run in parallel, the API handler breaks and skips applying about half of the new policies.<br></div><div>- bug where policies with > 2 replicas don't get properly auto-replicated is still present. Running rebalance_cluster.sh multiple times causes many queues to end up under-replicated after the first pass. however, there seems to be a workaround: if I apply a global replication policy of "exactly 3", and then delete the per-queue policies, this seems to force a resync as it switches queues to the global policy, and so they end up fully replicated to the 3 nodes I specified with the per-queue policies.<br>
</div><div>- still lots of replication problems related to pulling and re-adding nodes. running the toggle_nodes.sh script in a loop with the global "exactly 3" policy demonstrates that as nodes are removed, it doesn't automatically replicate to a different node, just leaves queues with only 1 replica, and doesn't always re-replicate back to the node which was pulled after it's been re-added. after a few iterations (3-5), this causes data loss on queues, with queues that have sequentially lost all their replicas ending up empty, even though the global policy says they should be ensuring there's always 3 copies of the data.<br>
</div><div>- running toggle_nodes.sh in a loop will also still get to the point where rabbitmq refuses to do any further node leave/join operations until the cluster is fully killed and brought back up.<br></div><div>- this also eventually causes a few of the queues to enter the weird state where they list unsyncronized mirrors, but aren't attempting to sync. at this point, rebalance_cluster.sh will also fail, and the cluster will reject all policy operation requests, until it's fully killed and restarted.<br>
<br></div><div>On the bright side:<br><br></div><div>- loading 10k messages per queue seems faster on my VM than with the 3.2.0 release, even when doing 3x local replication.<br></div><div>- bug where it was deleting queues as they approached 10k messages when policies had been applied seems to be gone.<br>
</div>- cluster management functions run sequentially from scripts seems to be safe now, haven't been able to generate any bad behaviour as long as I'm not running more than one cluster command at a time, and as long as I'm not removing and adding nodes.<br>
<div>- have a known good way to rebalance queues across cluster systems, even if it is slow and requires a workaround to get fully synced.<br></div><br></div>I'm really hoping you guys can replicate the more severe errors around polices and node management. I'm happy to provide thorough reproduction steps, information, logs, etc for anything I've reported here, if you guys are having issues reproducing internally.<br>
<div><div><br></div><div>Graeme<br><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Nov 4, 2013 at 11:00 AM, Graeme N <span dir="ltr"><<a href="mailto:graeme@sudo.ca" target="_blank">graeme@sudo.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Simon,<br><br>It does seem likely to me that fixing the initial issues I found would resolve the other problems, so I'll definitely re-test within the next day or so using the latest nightly build, and let you know how it goes.<br>
<br>Thanks!<span class="HOEnZb"><font color="#888888"><br>Graeme<br></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Nov 4, 2013 at 6:05 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On 29/10/2013 12:06AM, Graeme N wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
These errors are all really strange, and so I'm hoping you guys can<br>
reproduce them, and best case scenario, find something that accounts for<br>
these problems which we can then patch in our production environment.<br>
</blockquote>
<br></div>
We haven't been able to recreate your most recent issues - but it's possible that they're all copies of the same underlying problem as the first lot you reported (which are now fixed).<br>
<br>
So would you be able to have another go, with a recent nightly build?<br>
<br>
<a href="http://www.rabbitmq.com/nightly-builds.html" target="_blank">http://www.rabbitmq.com/<u></u>nightly-builds.html</a><div><div><br>
<br>
Cheers, Simon<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>