<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><br>
Could you describe accurately what your tests are so that we can try and<br>
reproduce?<br>
<div><div></div><div class="h5"><br></div></div></blockquote><div><br></div><div>I'm afraid the tests are of the system overall, and I haven't been able to produce a minimal test case so far. This evening's test involved dropping backed up messages into the queues quickly, while draining them slowly. Effectively: </div>
<div><br></div><div>5 HA queues, replicated to all nodes. 3 nodes in the cluster, 15GB machines, high water mark at 5.9GB.</div><div><br></div><div>Start producers and consumers, consumers running slowly, allowing queues to build to 500k.</div>
<div><span style="background-color: transparent; "><br></span></div><div><span style="background-color: transparent; ">Stop producers and consumers, delete queues. Deletions take a long time. Although I've also seen this with no-HA queues - it can take tens of minutes to delete a queue with 250k+ messages in.</span></div>
<div><span style="background-color: transparent; "><br></span></div><div><span style="background-color: transparent; ">Previously we've had queues in a steady state, with approximately 25,000 unacked messages (they take several minutes to process, aren't acked until complete). Then</span><span style="background-color: transparent; "> kill some nodes off, forcing the messages to be requeued and replayed on the slaves. It all gets out of sync after that.</span></div>
<div><span style="background-color: transparent; "><br></span></div><div>I might be able to give you a better test case once we've pushed our non-HA release out and I have a bit more time.</div></div>