[rabbitmq-discuss] Distributing Tasks to Worker Queues Across Federated Exchanges
matthias at rabbitmq.com
Fri Dec 21 19:34:02 GMT 2012
On 21/12/12 16:59, Chris wrote:
> Thanks for the reply, Tim. The nodes may be thousands of miles apart
> and each geographic location will have its own cluster as well. We're
> trying to find a solution that doesn't require any additional hardware
> and can be configured somewhat easily via a GUI. A load balancer also
> would not understand how to properly route based on queues/exchanges--
> the key is only to distribute/load-balance over the nodes that actually
> have consumers for the given exchange/queue.
It is not quite clear what problem you are trying to solve. In your
original email you said:
Broker A: Workers listening on queues with routing keys X & Y
Broker B: Workers listening on queues with routing keys Y & Z
1) Submit job with routing key X from Broker A: Should go to Consumer of
X on Broker A
2) Submit job with routing key Y from Broker A: Should go ONLY to
Consumer of Y on Broker A
3) Submit job with routing key Z from Broker A: Should go to Consumer of
Z on Broker A
Re (3) - there is no consumer of Z on A. I'm guessing you meant B.
Overall, I am guessing what you are trying to say here is that you have
three types of task - X,Y,Z - and workers that know how to process a
sub-set of these tasks, and that these workers might be in different
locations. And each task should be processed by only one worker.
So why not simply have a single broker, in one location, with three
queues - X, Y, Z? Workers, wherever they are, connect to the broker and
start consuming from the queues whose tasks they can handle.
One step up from this simple set up would be a federation where the X,
Y, Z queues reside on different brokers in the federation. The main
complication that creates is that workers need to know which of the
federated brokers they need to connect to.
Beware of premature optimisations. I'd stick with the simple setup
unless you have determined experimentally that it does not perform
adequately for your use case.
More information about the rabbitmq-discuss