[rabbitmq-discuss] Distributing Tasks to Worker Queues Across Federated Exchanges

[Chris]

Hmmm... I just realized that the proposed configuration of setting things
up in a ring will have one other adverse affect (besides weakening HA).

If I have the ring setup like this:  A --> B --> C --> A... then if B and C
both have worker queues for the same type of task, B will tend to get too
much preference because it will handle all requests from A and B (since the
requests will never go to the AE to route to C)-- whereas the ideal setup
would equally distribute requests from A to B and to C.  Hmmm...

Is there any ongoing work to make RabbitMQ support a feature more like
ActiveMQ's network of brokers?  In ActiveMQ's network of brokers, I think
it truly operates as one logical bus (still giving preference to local
consumers)-- which seems more like what I want.  (That said, I have heard a
few horror stories about ActiveMQ's network of brokers too.)

Thanks,
Chris


On Tue, Dec 18, 2012 at 12:13 PM, Chris <stuff at moesel.net> wrote:

> Hi Simon,
>
> Thanks for the suggestions.  This definitely gets me closer to what I'm
> looking for!  I didn't know about the alternate-exchange, so that's very
> helpful to know (and could be helpful in other scenarios too).  The
> ring-configuration sounds like it could work too, but then I guess I am
> giving up some high availability-- in that a single broken link breaks the
> chain and potentially prevents communication to other brokers that are
> still running fine.
>
> Definitely something to think about.  Thanks, Simon!
>
> -Chris
>
>
> On Tue, Dec 18, 2012 at 10:10 AM, Simon MacMullen <simon at rabbitmq.com>wrote:
>
>> On 18/12/12 13:55, Chris wrote:
>>
>>> I have a federation of brokers (let's say two: A and B).  They are
>>> federated because they are across WAN, so I know clustering will not
>>> work.  I would like to setup worker queues so that if I submit a job to
>>> an exchange/queue, it will be processed by one and only one consumer
>>> across the federated brokers.  Ideally, it should also prefer local
>>> consumers before going to federated consumers.
>>>
>>> For example:
>>>
>>> Broker A: Workers listening on queues with routing keys X & Y
>>> Broker B: Workers listening on queues with routing keys Y & Z
>>>
>>> 1) Submit job with routing key X from Broker A: Should go to Consumer of
>>> X on Broker A
>>> 2) Submit job with routing key Y from Broker A: Should go ONLY to
>>> Consumer of Y on Broker A
>>> 3) Submit job with routing key Z from Broker A: Should go to Consumer of
>>> Z on Broker A
>>>
>>
>> I think that the alternate-exchange mechanism (
>> http://www.rabbitmq.com/ae.**html <http://www.rabbitmq.com/ae.html>)
>> might help you here. Publish to a non-federated direct exchange X1. Declare
>> X1 with an AE of a federated direct exchange: X2.
>>
>> Your workers will need to bind to both X1 and X2 unfortunately, but
>> messages should get routed the way you want - if there's a local binding
>> then the message never hits the federated exchange, but if there isn't it
>> then goes out over federation.
>>
>> What this *won't* do is ensure the message only gets delivered to *one*
>> other broker over federation. Aha! But you could do that by making each X1
>> be federated, but have its upstream set to a single remote X2, with the
>> broker federation links arranged in a ring. Then messages go round the
>> ring, at each node either being delivered locally or forwarded on by
>> federation. You'd need to set max_hops to the ring size at each point, so
>> that unroutable messages go round the ring once and then die.
>>
>> Does this make sense?
>>
>> Cheers, Simon
>>
>> --
>> Simon MacMullen
>> RabbitMQ, VMware
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121221/003a8014/attachment.htm>