[rabbitmq-discuss] Distributing Tasks to Worker Queues Across Federated Exchanges

Fri Dec 21 20:08:10 GMT 2012

Hi Matthias,

Thanks for your reply.  I too originally preferred to go with the simpler
setup of having a single broker (or cluster of brokers).  Since the
services and clients will literally be distributed across the world,
however, our product owner did not want all traffic to have to go over the
WAN (in some cases volume may be high).  In addition, each geographic
location needs to be able to continue to operate in a basic fashion if it
becomes disconnected from the WAN (with the understanding that the set of
available services may be smaller during the disconnected period).

You're right about #3 though-- a typo on my part!

Thanks again for the help-- I think I have a basic understanding of my
options now...

-Chris

On Fri, Dec 21, 2012 at 2:34 PM, Matthias Radestock
<matthias at rabbitmq.com>wrote:

> Chris,
>
>
> On 21/12/12 16:59, Chris wrote:
>
>> Thanks for the reply, Tim.  The nodes may be thousands of miles apart
>> and each geographic location will have its own cluster as well.  We're
>> trying to find a solution that doesn't require any additional hardware
>> and can be configured somewhat easily via a GUI.  A load balancer also
>> would not understand how to properly route based on queues/exchanges--
>> the key is only to distribute/load-balance over the nodes that actually
>> have consumers for the given exchange/queue.
>>
>
> It is not quite clear what problem you are trying to solve. In your
> original email you said:
>
> <quote>
>
> Broker A: Workers listening on queues with routing keys X & Y
> Broker B: Workers listening on queues with routing keys Y & Z
>
> 1) Submit job with routing key X from Broker A: Should go to Consumer of X
> on Broker A
> 2) Submit job with routing key Y from Broker A: Should go ONLY to Consumer
> of Y on Broker A
> 3) Submit job with routing key Z from Broker A: Should go to Consumer of Z
> on Broker A
> </quote>
>
> Re (3) - there is no consumer of Z on A. I'm guessing you meant B.
>
> Overall, I am guessing what you are trying to say here is that you have
> three types of task - X,Y,Z - and workers that know how to process a
> sub-set of these tasks, and that these workers might be in different
> locations. And each task should be processed by only one worker.
>
> So why not simply have a single broker, in one location, with three queues
> - X, Y, Z? Workers, wherever they are, connect to the broker and start
> consuming from the queues whose tasks they can handle.
>
> One step up from this simple set up would be a federation where the X, Y,
> Z queues reside on different brokers in the federation. The main
> complication that creates is that workers need to know which of the
> federated brokers they need to connect to.
>
> Beware of premature optimisations. I'd stick with the simple setup unless
> you have determined experimentally that it does not perform adequately for
> your use case.
>
> Regards,
>
> Matthias
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121221/6766e534/attachment.htm>