<div dir="ltr"><div class="gmail_extra" style="font-family:arial,sans-serif;font-size:13px"><div class="gmail_quote"><div class="im">On Tue, May 13, 2014 at 3:59 PM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
</div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">On 13/05/2014 14:25, mouad ben wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">RabbitMQ cluster (RabbitMQ 3.1.5,Erlang R14B04)<br></blockquote>
<div><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">(Queues are also created with x-ha-policy set to all).<br>
</blockquote><br></div>As an aside, note that x-ha-policy only controls queue mirroring in 2.x! In 3.x it does nothing at all. See<a href="http://www.rabbitmq.com/ha.html" target="_blank">http://www.rabbitmq.com/ha.<u></u>html</a> and <a href="http://www.rabbitmq.com/blog/2012/11/19/breaking-things-with-rabbitmq-3-0/" target="_blank">http://www.rabbitmq.com/blog/<u></u>2012/11/19/breaking-things-<u></u>with-rabbitmq-3-0/</a></blockquote>
<div><br></div></div><div>Ohh thanks for the info, shamelessly i didn't knew that.</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br><br>Of course, autodelete RPC queues quite possibly do not need to be HA.</blockquote><div><br></div></div><div>Well apparently my "chaos monkey" wasn't able to reproduce this problem when mirroring queue was enabled !? I am actually still not sure why is this the case !? I am probably missing something obvious !</div>
<div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
But in the same time if a node of the cluster go down the Queue consumer<br>that are created in this node will be deleted, the Queues with<br>auto-delete will end up being deleted too and the same thing with the<br>exchanges bounded to them which also have auto-delete, all of this will<br>
be done **eventually** in all RabbitMQ nodes that are still up.<br><br>So in detail from neutron side we have:<br><br>N1. Connect to node 2.<br>N2. Create Exchange X.<br>N3. Create Queue Q.<br>N4. Create Binding from Q to X.<br>
<br> From cluster side we have:<br><br>R1. Delete consumer.<br>R2. Delete Queue Q (Binding is deleted explicitly).<br>R3. Delete Exchange X.<br><br>This actually can lead to a race condition that will result of N4<br>failing with error stating that exchange doesn't exist because<br>
apparently R3 action was executed after N2 and before N4.<br></blockquote><br></div>That makes sense. But note that R2 and R3 are an atomic event.</blockquote><div><br></div></div><div>Just to be sure, you don't mean that (R2+R3) group is an atomic event !?</div>
<div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
A workaround that we have created is to retry creation if it fail as you<br>can see in the bug report in OpenStack side<br><a href="https://bugs.launchpad.net/neutron/+bug/1318721" target="_blank">https://bugs.launchpad.net/<u></u>neutron/+bug/1318721</a>.<br>
</blockquote><br></div>And that would work as a workaround.<div><br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
But i think that this also a problem in RabbitMQ side, basically i<br>believe a more sane behavior will be for RabbitMQ to ignore **old**<br>delete if a newest exchange's declare was sent, instead of threading<br>this later as a no op.<br>
</blockquote><br></div>Hmm. Newer than what? The queue delete R2 is not necessarily older than then most recent exchange declare N2 in this scenario, is it? In fact, isn't R2 always newer than N2?<br></blockquote><div>
<br></div></div><div>Not necessarily, I don't see why there should be any order between unrelated operation from client versus cluster side, when the connection is broken this will raise the sequence of events in both sides but who is faster ! in the "happy path" the order will be R1 -> R2 -> R3 -> N1 -> N2 -> N3 -> N4, but in reality all orders are possible e.g. R1 -> N1 -> N2 -> R2 -> N3 -> R3 -> N4, right ? BTW when i say order, i am taking as reference the RabbitMQ cluster b/c operation are atomics right ?</div>
<div class="im"><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>(Also, doing this would require timestamps in queues, exchanges and bindings, updated cluster-wide every time they are re-declared. That would be a big problem for performance.)</blockquote>
<div><br></div></div><div>Agree, timestamp are surely not that easy to manage in a distributed environment b/c they require the clocks in the different nodes to be in sync, vector clock may be another solution but well i am not well versed enough in distributed system to give any real feedback on this, that was just a dummy idea from my part anyway :) </div>
<div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Does my analyze above make sense !?<br></blockquote><br></div>So I'm not sure it does.<br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>When reading also the AMQP 0-9-1 reference<br>(<a href="https://www.rabbitmq.com/amqp-0-9-1-reference.html" target="_blank">https://www.rabbitmq.com/<u></u>amqp-0-9-1-reference.html</a>) i found that:<br><br></div> * The server SHOULD allow for a reasonable delay between the point<div>
<br> when it determines that an exchange is not being used (or no longer<br> used), and the point when it deletes the exchange. At the least it<br> must allow a client to create an exchange and then bind a queue to<br>
it, with a small but non-zero delay between these two actions.<br><br><br>Does my finding contradict this ? And how big is the delay ?<br></div></blockquote><br>It does; there is no delay in RabbitMQ.<br><br>Note that if there were a delay then you'd still have exactly the same race anyway, it wouldn't solve anything.<br>
<br>You can get a similar delay for the queue deletion, by using x-expires instead of auto-delete. But once the queue goes, the bindings and exchange all go atomically.<br><br>So how to solve your problem? It sounds like each thing that can receive RPC replies declares an exchange, queue and binding for that purpiose, with the exchange and queue not linked in to anything else. Is that correct?<br>
</blockquote><div><br></div></div><div>Yes that's the main idea of how OpenStack implement RPC.</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>If that is so, why not only declare a queue, and have the RPC server publish to that queue directly? The queue could still be auto-delete, and everything would be simpler. Or is there some subtlety I'm not getting?</blockquote>
<div><br></div></div><div>You mean by publishing to the "nameless exchanger", that sounds much easier, but i am not why that is not the case i need probably to ask in the OpenStack mailing list first.</div><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
One last thing is the auto-delete deprecation from<br><a href="http://www.rabbitmq.com/amqp-0-9-1-errata.html" target="_blank">http://www.rabbitmq.com/amqp-<u></u>0-9-1-errata.html</a>, point 25:<br><br>The 'auto-delete' flag on 'exchange.declare' got deprecated in 0-9-1.<br>
Auto-delete exchanges are actually quite useful, so this flag should be<br>restored.<br><br>Does this mean the auto-delete flag will not be removed from RabbitMQ or<br>what ?<br></blockquote><br></div>It means we will continue to support it, yes.<br>
<br>Cheers, Simon<br></blockquote></div></div><br></div><div class="gmail_extra" style="font-family:arial,sans-serif;font-size:13px">Thanks for the quick answer too, and sorry for answering you directly instead of answering to the mailing list, dummy mistake from my part :(</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, May 13, 2014 at 3:59 PM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 13/05/2014 14:25, mouad ben wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
RabbitMQ cluster (RabbitMQ 3.1.5,Erlang R14B04)<br>
</blockquote><div class="">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
(Queues are also created with x-ha-policy set to all).<br>
</blockquote>
<br></div>
As an aside, note that x-ha-policy only controls queue mirroring in 2.x! In 3.x it does nothing at all. See <a href="http://www.rabbitmq.com/ha.html" target="_blank">http://www.rabbitmq.com/ha.<u></u>html</a> and <a href="http://www.rabbitmq.com/blog/2012/11/19/breaking-things-with-rabbitmq-3-0/" target="_blank">http://www.rabbitmq.com/blog/<u></u>2012/11/19/breaking-things-<u></u>with-rabbitmq-3-0/</a><br>
<br>
Of course, autodelete RPC queues quite possibly do not need to be HA.<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
But in the same time if a node of the cluster go down the Queue consumer<br>
that are created in this node will be deleted, the Queues with<br>
auto-delete will end up being deleted too and the same thing with the<br>
exchanges bounded to them which also have auto-delete, all of this will<br>
be done **eventually** in all RabbitMQ nodes that are still up.<br>
<br>
So in detail from neutron side we have:<br>
<br>
N1. Connect to node 2.<br>
N2. Create Exchange X.<br>
N3. Create Queue Q.<br>
N4. Create Binding from Q to X.<br>
<br>
From cluster side we have:<br>
<br>
R1. Delete consumer.<br>
R2. Delete Queue Q (Binding is deleted explicitly).<br>
R3. Delete Exchange X.<br>
<br>
This actually can lead to a race condition that will result of N4<br>
failing with error stating that exchange doesn't exist because<br>
apparently R3 action was executed after N2 and before N4.<br>
</blockquote>
<br></div>
That makes sense. But note that R2 and R3 are an atomic event.<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
A workaround that we have created is to retry creation if it fail as you<br>
can see in the bug report in OpenStack side<br>
<a href="https://bugs.launchpad.net/neutron/+bug/1318721" target="_blank">https://bugs.launchpad.net/<u></u>neutron/+bug/1318721</a>.<br>
</blockquote>
<br></div>
And that would work as a workaround.<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
But i think that this also a problem in RabbitMQ side, basically i<br>
believe a more sane behavior will be for RabbitMQ to ignore **old**<br>
delete if a newest exchange's declare was sent, instead of threading<br>
this later as a no op.<br>
</blockquote>
<br></div>
Hmm. Newer than what? The queue delete R2 is not necessarily older than then most recent exchange declare N2 in this scenario, is it? In fact, isn't R2 always newer than N2?<br>
<br>
(Also, doing this would require timestamps in queues, exchanges and bindings, updated cluster-wide every time they are re-declared. That would be a big problem for performance.)<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Does my analyze above make sense !?<br>
</blockquote>
<br></div>
So I'm not sure it does.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">
When reading also the AMQP 0-9-1 reference<br>
(<a href="https://www.rabbitmq.com/amqp-0-9-1-reference.html" target="_blank">https://www.rabbitmq.com/<u></u>amqp-0-9-1-reference.html</a>) i found that:<br>
<br></div>
* The server SHOULD allow for a reasonable delay between the point<div class=""><br>
when it determines that an exchange is not being used (or no longer<br>
used), and the point when it deletes the exchange. At the least it<br>
must allow a client to create an exchange and then bind a queue to<br>
it, with a small but non-zero delay between these two actions.<br>
<br>
<br>
Does my finding contradict this ? And how big is the delay ?<br>
</div></blockquote>
<br>
It does; there is no delay in RabbitMQ.<br>
<br>
Note that if there were a delay then you'd still have exactly the same race anyway, it wouldn't solve anything.<br>
<br>
You can get a similar delay for the queue deletion, by using x-expires instead of auto-delete. But once the queue goes, the bindings and exchange all go atomically.<br>
<br>
So how to solve your problem? It sounds like each thing that can receive RPC replies declares an exchange, queue and binding for that purpiose, with the exchange and queue not linked in to anything else. Is that correct?<br>
<br>
If that is so, why not only declare a queue, and have the RPC server publish to that queue directly? The queue could still be auto-delete, and everything would be simpler. Or is there some subtlety I'm not getting?<div class="">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
One last thing is the auto-delete deprecation from<br>
<a href="http://www.rabbitmq.com/amqp-0-9-1-errata.html" target="_blank">http://www.rabbitmq.com/amqp-<u></u>0-9-1-errata.html</a>, point 25:<br>
<br>
The 'auto-delete' flag on 'exchange.declare' got deprecated in 0-9-1.<br>
Auto-delete exchanges are actually quite useful, so this flag should be<br>
restored.<br>
<br>
Does this mean the auto-delete flag will not be removed from RabbitMQ or<br>
what ?<br>
</blockquote>
<br></div>
It means we will continue to support it, yes.<br>
<br>
Cheers, Simon<br>
</blockquote></div><br></div>