[rabbitmq-discuss] auto-delete queue/exchanges and cluster synchronization

[Simon MacMullen]

On 16/05/14 10:42, mouad ben wrote:
> On Tue, May 13, 2014 at 3:59 PM, Simon MacMullen <simon at rabbitmq.com
>     Of course, autodelete RPC queues quite possibly do not need to be HA.
>
> Well apparently my "chaos monkey" wasn't able to reproduce this problem
> when mirroring queue was enabled !? I am actually still not sure why is
> this the case !? I am probably missing something obvious !

Possibly. Of course since this is a race anyway, the timings might have 
just changed. Especially since deleting an HA queue is quite a bit 
slower than deleting a non-HA one for obvious reasons.

>         But in the same time if a node of the cluster go down the Queue
>         consumer
>         that are created in this node will be deleted, the Queues with
>         auto-delete will end up being deleted too and the same thing
>         with the
>         exchanges bounded to them which also have auto-delete, all of
>         this will
>         be done **eventually** in all RabbitMQ nodes that are still up.
>
>         So in detail from neutron side we have:
>
>         N1. Connect to node 2.
>         N2. Create Exchange X.
>         N3. Create Queue Q.
>         N4. Create Binding from Q to X.
>
>           From cluster side we have:
>
>         R1. Delete consumer.
>         R2. Delete Queue Q (Binding is deleted explicitly).
>         R3. Delete Exchange X.
>
>         This actually can lead to a race condition that will result of N4
>         failing with error stating that exchange doesn't exist because
>         apparently R3 action was executed after N2 and before N4.
>
>
>     That makes sense. But note that R2 and R3 are an atomic event.
>
>
> Just to be sure, you don't mean that (R2+R3) group is an atomic event !?

I'm assuming here that "Binding is deleted explicitly" is a typo for 
"Binding is deleted implicitly" - that is, you explicitly delete the 
queue, which implicitly deletes the binding and thus auto-deletes the 
exchange. If so, then yes, that's atomic.

Or does your code explicitly delete the queue and the exchange? If so, 
the exchange.delete AMQP method will be ignored since deleting a 
non-existent exchange is now a no-op.

Sorry, I probably didn't explain that very clearly the first time round. 
Hope it's clearer now.

>         But i think that this also a problem in RabbitMQ side, basically i
>         believe a more sane behavior will be for RabbitMQ to ignore **old**
>         delete if a newest exchange's declare was sent, instead of threading
>         this later as a no op.
>
>     Hmm. Newer than what? The queue delete R2 is not necessarily older
>     than then most recent exchange declare N2 in this scenario, is it?
>     In fact, isn't R2 always newer than N2?
>
> Not necessarily, I don't see why there should be any order between
> unrelated operation from client versus cluster side, when the connection
> is broken this will raise the sequence of events in both sides but who
> is faster ! in the "happy path" the order will be R1 -> R2 -> R3 -> N1
> -> N2 -> N3 -> N4, but in reality all orders are possible e.g. R1 -> N1
> -> N2 -> R2 -> N3 -> R3 -> N4, right ?

Sorry, again I was unclear. When I said "in this scenario" I meant "in 
the scenario where the breakage that you're observing occurs". Yes, 
other interleavings are of course possible :-)

> BTW when i say order, i am taking
> as reference the RabbitMQ cluster b/c operation are atomics right ?

Yes.

>     So how to solve your problem? It sounds like each thing that can
>     receive RPC replies declares an exchange, queue and binding for that
>     purpiose, with the exchange and queue not linked in to anything
>     else. Is that correct?
>
> Yes that's the main idea of how OpenStack implement RPC.
>
>     If that is so, why not only declare a queue, and have the RPC server
>     publish to that queue directly? The queue could still be
>     auto-delete, and everything would be simpler. Or is there some
>     subtlety I'm not getting?
>
> You mean by publishing to the "nameless exchanger", that sounds much
> easier, but i am not why that is not the case i need probably to ask in
> the OpenStack mailing list first.

Yes, that was what I meant.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal