[rabbitmq-discuss] Behaviour of Federated Queues

[Josh West]

On 01/24/2014 07:20 AM, Simon MacMullen wrote:
>
>> I have noticed that if I create a queue on broker1 with name
>> "broker1.foo.bar" (for example), the queue is automatically created on
>> broker2 and broker3.  Should this occur?  Or should federation only
>> create the queue on upstream brokers when needed (ad-hoc)?
>
> It should occur. I'm not sure when 'needed' would be; when you publish 
> to it?
>
> If your aim is to make sure that you can publish anywhere and have the 
> publishing application determine which node the message ends up on, 
> you might find that exchange federation works better for you; have a 
> single federated exchange and bind (unfederated) queues to it with 
> routing keys like broker1.*.

Yeah, this is a good idea.  This is actually the first route I 
attempted.  The issue I ran into was attempting RPC across federation -- 
where the callback queue may not be propagated in time.  Using federated 
queues solved the problem as long as the RPC callback queue begins with 
brokerN, where brokerN is the original caller.  This way I don't need to 
wait for the asynchronous federation.  The RPC server creates the 
reply_to queue on their end and of course the RPC client has also 
created that callback queue. As long as the names match up, whenever the 
async federation takes place, the message(s) in the callback queue on 
the RPC server side will zip on over to the callback queue on the RPC 
client side.  Very cool stuff :-)

>
>> 2. When queues are created with the Auto Delete parameter, that
>> parameter is of course properly propagated.  But say a client puts a
>> message into the queue "broker1.foo.bar" on broker2 and say a client
>> retrieves the message from that same queue on broker1.  The Auto Delete
>> parameter behaviour would be triggered and the queue is then deleted
>> from broker1 and broker2 (upon client disconnection). But... Since
>> nobody has ever put any messages into the queue "broker1.foo.bar"
>> residing on broker3, the queue will remain defined possibly 
>> indefinitely.
>
> That's not how autodelete works - queues autodelete when their last 
> consumer goes away, regardless of whether messages have been published 
> to them.
>
> So I see this working correctly - when a consumer consumes from 
> broker1.foo.bar on broker1, that consumer gets propagated to 
> broker1.foo.bar on broker3. When the consumer is cancelled, cancel is 
> also propagated and both queues delete.

I would think this would be the case.  But I believe I have found a 
situation where this is only happening properly 80% of the time.

I have modified the RPC code example from the rabbitmq tutorial. The 
modifications are such that the RPC server will also define the queue on 
their end with queue_declare().  Additionally, the RPC client chooses an 
explicit name for the RPC callback queue instead of an automatically 
assigned one.  The naming format is <broker>.<uuid> (e.g. 
broker2.3e815ef7-1322-49f1-8ca8-14118b4308df) if the RPC client is 
making the request from broker2.  If I run the RPC client 100 times, the 
code works all 100 times.  However keep in mind that there are 3 brokers 
in the federation scheme here, but broker1 and broker2 are the only ones 
communicating in this RPC example.  The <broker>.<uuid> queues are still 
declared/propagated over to broker3.  But only 80% of the time are the 
deletions also propagated.  Thus if I run the RPC client 100 times, with 
broker2 making an RPC fib(n) request to a server running on broker1, the 
RPC callback queue remains on broker3 20% of the time and is never deleted.

>
> However, I think there is the opposite problem around auto-delete and 
> queue federation - since the federation mechanism controls message 
> flow by consuming and cancelling from the upstream, it could cause 
> federated autodelete queues to delete themselves too early.

Interesting.

>
> I'm not sure what, if anything, can be done about that. It might be 
> that autodelete and queue federation just don't work well together.
>

It may be more than an issue just with auto delete.  It looks like 
deletions themselves may not be propagated at all.  For example:


#!/usr/bin/python

import pika

credentials = pika.PlainCredentials("guest", "guest")
conn_params = pika.ConnectionParameters("localhost", credentials = 
credentials)
connection  = pika.BlockingConnection(conn_params)
channel     = connection.channel()

for i in range(1,100):
         queue = "broker1.queue.%i" % (i)
         channel.queue_declare(queue=queue, auto_delete=True)
         channel.queue_delete(queue=queue)

channel.close()
connection.close()


While the queues are deleted from broker1, they still remain on broker2 
and broker3 :-(

Josh West
One.com - http://www.one.com