[rabbitmq-discuss] Distributing Tasks to Worker Queues Across Federated Exchanges

Tue Dec 18 15:10:40 GMT 2012

On 18/12/12 13:55, Chris wrote:
> I have a federation of brokers (let's say two: A and B).  They are
> federated because they are across WAN, so I know clustering will not
> work.  I would like to setup worker queues so that if I submit a job to
> an exchange/queue, it will be processed by one and only one consumer
> across the federated brokers.  Ideally, it should also prefer local
> consumers before going to federated consumers.
>
> For example:
>
> Broker A: Workers listening on queues with routing keys X & Y
> Broker B: Workers listening on queues with routing keys Y & Z
>
> 1) Submit job with routing key X from Broker A: Should go to Consumer of
> X on Broker A
> 2) Submit job with routing key Y from Broker A: Should go ONLY to
> Consumer of Y on Broker A
> 3) Submit job with routing key Z from Broker A: Should go to Consumer of
> Z on Broker A

I think that the alternate-exchange mechanism 
(http://www.rabbitmq.com/ae.html) might help you here. Publish to a 
non-federated direct exchange X1. Declare X1 with an AE of a federated 
direct exchange: X2.

Your workers will need to bind to both X1 and X2 unfortunately, but 
messages should get routed the way you want - if there's a local binding 
then the message never hits the federated exchange, but if there isn't 
it then goes out over federation.

What this *won't* do is ensure the message only gets delivered to *one* 
other broker over federation. Aha! But you could do that by making each 
X1 be federated, but have its upstream set to a single remote X2, with 
the broker federation links arranged in a ring. Then messages go round 
the ring, at each node either being delivered locally or forwarded on by 
federation. You'd need to set max_hops to the ring size at each point, 
so that unroutable messages go round the ring once and then die.

Does this make sense?

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware