[rabbitmq-discuss] Distributing Tasks to Worker Queues Across Federated Exchanges
stuff at moesel.net
Fri Dec 21 15:20:28 GMT 2012
Hmmm... I just realized that the proposed configuration of setting things
up in a ring will have one other adverse affect (besides weakening HA).
If I have the ring setup like this: A --> B --> C --> A... then if B and C
both have worker queues for the same type of task, B will tend to get too
much preference because it will handle all requests from A and B (since the
requests will never go to the AE to route to C)-- whereas the ideal setup
would equally distribute requests from A to B and to C. Hmmm...
Is there any ongoing work to make RabbitMQ support a feature more like
ActiveMQ's network of brokers? In ActiveMQ's network of brokers, I think
it truly operates as one logical bus (still giving preference to local
consumers)-- which seems more like what I want. (That said, I have heard a
few horror stories about ActiveMQ's network of brokers too.)
On Tue, Dec 18, 2012 at 12:13 PM, Chris <stuff at moesel.net> wrote:
> Hi Simon,
> Thanks for the suggestions. This definitely gets me closer to what I'm
> looking for! I didn't know about the alternate-exchange, so that's very
> helpful to know (and could be helpful in other scenarios too). The
> ring-configuration sounds like it could work too, but then I guess I am
> giving up some high availability-- in that a single broken link breaks the
> chain and potentially prevents communication to other brokers that are
> still running fine.
> Definitely something to think about. Thanks, Simon!
> On Tue, Dec 18, 2012 at 10:10 AM, Simon MacMullen <simon at rabbitmq.com>wrote:
>> On 18/12/12 13:55, Chris wrote:
>>> I have a federation of brokers (let's say two: A and B). They are
>>> federated because they are across WAN, so I know clustering will not
>>> work. I would like to setup worker queues so that if I submit a job to
>>> an exchange/queue, it will be processed by one and only one consumer
>>> across the federated brokers. Ideally, it should also prefer local
>>> consumers before going to federated consumers.
>>> For example:
>>> Broker A: Workers listening on queues with routing keys X & Y
>>> Broker B: Workers listening on queues with routing keys Y & Z
>>> 1) Submit job with routing key X from Broker A: Should go to Consumer of
>>> X on Broker A
>>> 2) Submit job with routing key Y from Broker A: Should go ONLY to
>>> Consumer of Y on Broker A
>>> 3) Submit job with routing key Z from Broker A: Should go to Consumer of
>>> Z on Broker A
>> I think that the alternate-exchange mechanism (
>> http://www.rabbitmq.com/ae.**html <http://www.rabbitmq.com/ae.html>)
>> might help you here. Publish to a non-federated direct exchange X1. Declare
>> X1 with an AE of a federated direct exchange: X2.
>> Your workers will need to bind to both X1 and X2 unfortunately, but
>> messages should get routed the way you want - if there's a local binding
>> then the message never hits the federated exchange, but if there isn't it
>> then goes out over federation.
>> What this *won't* do is ensure the message only gets delivered to *one*
>> other broker over federation. Aha! But you could do that by making each X1
>> be federated, but have its upstream set to a single remote X2, with the
>> broker federation links arranged in a ring. Then messages go round the
>> ring, at each node either being delivered locally or forwarded on by
>> federation. You'd need to set max_hops to the ring size at each point, so
>> that unroutable messages go round the ring once and then die.
>> Does this make sense?
>> Cheers, Simon
>> Simon MacMullen
>> RabbitMQ, VMware
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss