[rabbitmq-discuss] Routing on multiple criteria but hitting only one consumer

Sun Apr 15 10:04:01 BST 2012

How can we distribute work such that worker pools can subscribe to
work messages based on differing (but frequently overlapping) criteria
from each other but such that when a message is routed that matches to
multiple worker pools, only one worker will pick up the job?

Simplified example:

* We have host1 and host2.
* Host1 handles jobs of classA and classB; host2 handles jobs of
classB and classC.
* If we route a job of classA, only host1 will pick it up; if we route
a job of classB, either host1 or host2 will pick it up (based on their
current load / first available) but never both.
It would seem that we need to use a topic exchange, as our routing
criteria is complex and using wildcards gives us the type of flexible
matching we want.

However:

1) If we use the same name for the worker pool queue (say “worker-
jobs”) we get the desired work splitting out to arbitrary matching
workers, but every worker subscribing to the named queue seems to
infect the other workers with each other’s routing criteria as they
bind it. I.e. the binding of the routing key seems to be at the
central queue name level not on a connection-to-queue basis.
2) If we use different queue names for each worker pool connection
(say “poolA-jobs” and “poolB-jobs”) to the same exchange then we get
the desired behavior with the different routing criteria maintained
between pools but a job coming in that can match to both poolA and
poolB gets routed to both of them (albeit only to one worker in each).

Notes:
1) I’ve spared you the details of why but suffice to say we have an
existing multi-petabyte distributed search application that needs
response times < 50ms. We already achieve this with our own custom
routing hub but we’d like to replace this with RabbitMQ as its
performance is attractive (as is retiring homemade code that overlaps
with general purpose community projects) if we can get the more
sophisticated routing we need.
2) We use Python
3) We actually have the luxury of already knowing upfront which hosts
are capable of serving which jobs, so we can do something like have
host1 subscribe to #.host1.# and host2 to #.host2.#. Then when we
route a classB message, we can give it a key of host1.host2 to
indicate which backends are acceptable for service. This simplifies
the routing rules but still doesn’t overcome the problem described.