[rabbitmq-discuss] Routing on multiple criteria but hitting only one consumer

Mon Apr 16 12:51:27 BST 2012

On 16/04/2012, at 7:58 PM, Emile Joubert wrote:
> Hi,
> 
> On 15/04/12 10:04, AgileZebra wrote:
> 
>> * We have host1 and host2.
>> * Host1 handles jobs of classA and classB; host2 handles jobs of
>> classB and classC.
>> * If we route a job of classA, only host1 will pick it up; if we route
>> a job of classB, either host1 or host2 will pick it up (based on their
>> current load / first available) but never both.
> 
> Can you constrain the routing so that messages are only routed to one
> queue, and you have one queue per class? Then host1 can subscribe to the
> queues for classA and classB, and host2 can subscribe to the queues for
> classB and classC.

Unfortunately not - there are tens of thousands of 'classes' and only tens of worker hosts. 
I tried to make the example fairly simple and generic but I guess I should have given you more real details. The classes are actually databases. There are databases for every hour of every day of every year, each with gigs and gigs of data, The databases are replicated in a semi-stripped, semi-random fashion to multiple nodes. They move around regularly depending on which data is hot and which is not, and in and out of SSD filesystems, or growing or shrinking their number of clones as required. We have a database that maps each task in a job to a list of locations where the database live and we want to allow the workers to take jobs as their workload dictates and naturally balance. We're already achieving this with our own broker but were really hoping to bypass it.  