[rabbitmq-discuss] Distributing Tasks to Worker Queues Across Federated Exchanges

Tue Dec 18 17:13:20 GMT 2012

Hi Simon,

Thanks for the suggestions.  This definitely gets me closer to what I'm
looking for!  I didn't know about the alternate-exchange, so that's very
helpful to know (and could be helpful in other scenarios too).  The
ring-configuration sounds like it could work too, but then I guess I am
giving up some high availability-- in that a single broken link breaks the
chain and potentially prevents communication to other brokers that are
still running fine.

Definitely something to think about.  Thanks, Simon!

-Chris

On Tue, Dec 18, 2012 at 10:10 AM, Simon MacMullen <simon at rabbitmq.com>wrote:

> On 18/12/12 13:55, Chris wrote:
>
>> I have a federation of brokers (let's say two: A and B).  They are
>> federated because they are across WAN, so I know clustering will not
>> work.  I would like to setup worker queues so that if I submit a job to
>> an exchange/queue, it will be processed by one and only one consumer
>> across the federated brokers.  Ideally, it should also prefer local
>> consumers before going to federated consumers.
>>
>> For example:
>>
>> Broker A: Workers listening on queues with routing keys X & Y
>> Broker B: Workers listening on queues with routing keys Y & Z
>>
>> 1) Submit job with routing key X from Broker A: Should go to Consumer of
>> X on Broker A
>> 2) Submit job with routing key Y from Broker A: Should go ONLY to
>> Consumer of Y on Broker A
>> 3) Submit job with routing key Z from Broker A: Should go to Consumer of
>> Z on Broker A
>>
>
> I think that the alternate-exchange mechanism (http://www.rabbitmq.com/ae.
> **html <http://www.rabbitmq.com/ae.html>) might help you here. Publish to
> a non-federated direct exchange X1. Declare X1 with an AE of a federated
> direct exchange: X2.
>
> Your workers will need to bind to both X1 and X2 unfortunately, but
> messages should get routed the way you want - if there's a local binding
> then the message never hits the federated exchange, but if there isn't it
> then goes out over federation.
>
> What this *won't* do is ensure the message only gets delivered to *one*
> other broker over federation. Aha! But you could do that by making each X1
> be federated, but have its upstream set to a single remote X2, with the
> broker federation links arranged in a ring. Then messages go round the
> ring, at each node either being delivered locally or forwarded on by
> federation. You'd need to set max_hops to the ring size at each point, so
> that unroutable messages go round the ring once and then die.
>
> Does this make sense?
>
> Cheers, Simon
>
> --
> Simon MacMullen
> RabbitMQ, VMware
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121218/85f6fc9a/attachment.htm>