<div dir="ltr">Hi all,<div><br></div><div>I would like to start by apologizing for the long email but i have quiet a few questions that i need to clarify :)<br><div><br></div><div>So about my problem, i would like to report a miss-behavior of the RabbitMQ cluster (<span style="text-align:right;color:rgb(68,68,68);font-family:Verdana,sans-serif;font-size:12px">RabbitMQ 3.1.5,</span><span style="text-align:right;color:rgb(68,68,68);font-family:Verdana,sans-serif;font-size:12px"> </span><acronym class="" title="Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:30] [kernel-poll:true]" style="text-align:right;color:inherit;font-family:Verdana,sans-serif;font-size:12px;background-image:none;padding:0px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px;border-style:none none dotted;border-bottom-width:1px">Erlang R14B04</acronym>) that we come to notice in our OpenStack cluster, that use RabbitMQ for message delivery. Understanding OpenStack is not required here but i will try to abstract what is that as much as i can.</div>
<div><br></div><div>Basically we have 3 nodes RabbitMQ cluster and a bunch OpenStack services that use RabbitMQ to communicate between them, one of the mean of communication is RPC, which is done by creating exchanges, queues with auto-delete flag set, which are the one that exhibit the problem.</div>
<div><br></div><div>So usually all OpenStack services start using one node of the cluster and when this node go down they connect to another node and make sure that all exchanges and queues are re-created there, usually the re-creation part end up being a no-op b/c of the cluster synchronization (Queues are also created with x-ha-policy set to all).</div>
<div><br></div><div>But in the same time if a node of the cluster go down the Queue consumer that are created in this node will be deleted, the Queues with auto-delete will end up being deleted too and the same thing with the exchanges bounded to them which also have auto-delete, all of this will be done **eventually** in all RabbitMQ nodes that are still up.</div>
<div><br></div><div>So in detail from neutron side we have:</div><div><br></div><div><span style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">N1. Connect to node 2.</span><br style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">
<span style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">N2. Create Exchange X.</span><br style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">
<span style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">N3. Create Queue Q.</span><br style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">
<span style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">N4. Create Binding from Q to X.</span><br></div><div><span style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px"><br>
</span></div><div>From cluster side we have:<br></div><div><br></div><div><span style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">R1. Delete consumer.</span><br style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">
<span style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">R2. Delete Queue Q (Binding is deleted explicitly).</span><br style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">
<span style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px">R3. Delete Exchange X.</span><br></div><div><span style="color:rgb(51,51,51);font-family:'Ubuntu Mono',monospace;font-size:12px;line-height:18px"><br>
</span></div><div><div>This actually can lead to a race condition that will result of N4 failing with error stating that exchange doesn't exist because apparently R3 action was executed after N2 and before N4.</div></div>
<div><br></div><div>A workaround that we have created is to retry creation if it fail as you can see in the bug report in OpenStack side <a href="https://bugs.launchpad.net/neutron/+bug/1318721">https://bugs.launchpad.net/neutron/+bug/1318721</a>. </div>
<div><br></div><div>But i think that this also a problem in RabbitMQ side, basically i believe a more sane behavior will be for RabbitMQ to ignore **old** delete if a newest exchange's declare was sent, instead of threading this later as a no op.</div>
<div><br></div><div>Does my analyze above make sense !?</div><div><br></div><div>When reading also the AMQP 0-9-1 reference (<a href="https://www.rabbitmq.com/amqp-0-9-1-reference.html">https://www.rabbitmq.com/amqp-0-9-1-reference.html</a>) i found that:</div>
<div><br></div><div><ul class="" style="margin:0px;padding:0px;color:rgb(85,85,85);font-family:Verdana,sans-serif;font-size:13px;line-height:18px"><li style="list-style-type:none;background-image:url(https://www.rabbitmq.com/img/li.gif);margin:0px;padding:6px 0px 0px 10px;background-repeat:no-repeat no-repeat">
The server SHOULD allow for a reasonable delay between the point when it determines that an exchange is not being used (or no longer used), and the point when it deletes the exchange. At the least it must allow a client to create an exchange and then bind a queue to it, with a small but non-zero delay between these two actions.</li>
</ul></div><div><br></div><div>Does my finding contradict this ? And how big is the delay ?<br></div><div><br></div><div>One last thing is the auto-delete deprecation from <a href="http://www.rabbitmq.com/amqp-0-9-1-errata.html">http://www.rabbitmq.com/amqp-0-9-1-errata.html</a>, point 25:</div>
<div><br></div><div> <span style="color:rgb(85,85,85);font-family:Verdana,sans-serif;font-size:13px;line-height:18px">The 'auto-delete' flag on 'exchange.declare' got deprecated in 0-9-1. Auto-delete exchanges are actually quite useful, so this flag should be restored.</span></div>
<div><br></div><div>Does this mean the auto-delete flag will not be removed from RabbitMQ or what ?<br></div><div><br></div><div>Thank you for you time and hope this was helpful<br></div><div><br></div><div>--</div><div>
Mouad Benchchaoui </div>
</div></div>