[rabbitmq-discuss] Distributing Tasks to Worker Queues Across Federated Exchanges

Fri Dec 21 19:34:02 GMT 2012

Chris,

On 21/12/12 16:59, Chris wrote:
> Thanks for the reply, Tim.  The nodes may be thousands of miles apart
> and each geographic location will have its own cluster as well.  We're
> trying to find a solution that doesn't require any additional hardware
> and can be configured somewhat easily via a GUI.  A load balancer also
> would not understand how to properly route based on queues/exchanges--
> the key is only to distribute/load-balance over the nodes that actually
> have consumers for the given exchange/queue.

It is not quite clear what problem you are trying to solve. In your 
original email you said:

<quote>
Broker A: Workers listening on queues with routing keys X & Y
Broker B: Workers listening on queues with routing keys Y & Z

1) Submit job with routing key X from Broker A: Should go to Consumer of 
X on Broker A
2) Submit job with routing key Y from Broker A: Should go ONLY to 
Consumer of Y on Broker A
3) Submit job with routing key Z from Broker A: Should go to Consumer of 
Z on Broker A
</quote>

Re (3) - there is no consumer of Z on A. I'm guessing you meant B.

Overall, I am guessing what you are trying to say here is that you have 
three types of task - X,Y,Z - and workers that know how to process a 
sub-set of these tasks, and that these workers might be in different 
locations. And each task should be processed by only one worker.

So why not simply have a single broker, in one location, with three 
queues - X, Y, Z? Workers, wherever they are, connect to the broker and 
start consuming from the queues whose tasks they can handle.

One step up from this simple set up would be a federation where the X, 
Y, Z queues reside on different brokers in the federation. The main 
complication that creates is that workers need to know which of the 
federated brokers they need to connect to.

Beware of premature optimisations. I'd stick with the simple setup 
unless you have determined experimentally that it does not perform 
adequately for your use case.

Regards,

Matthias