[rabbitmq-discuss] Round-robin exchanges?

Tue Oct 11 02:48:45 BST 2011

Hello,
I am trying to implement a work queue in RabbitMQ.  I have several machines
that will act as worker pools for consuming my queues.  I would like each
job to be processed only once.  Some jobs have environmental requirements,
ie must be executed on a machine with an SSD.  Not all worker pools will
meet these requirements.  A first approach would be to have a queue for
every permutation of requirements: "No Requirements", "Requires SSD",
"Requires Certificate", etc and have each worker pool subscribe to all
queues which it can handle.

A majority of the jobs will have no requirements, so many worker pools will
be underutilized.  To fight this, I am hoping to distribute jobs in a
round-robin fashion to all queues that can handle them.  For example, a
no-requirements job can be handled by any worker pool, and therefore should
be distributed equally to all of them.  That is, jobs w/ no requirements
will be evenly distributed to "No Requirements", "Requires SSD", "Requires
Certificate", etc.

I tried this with topic exchanges, but of course they will distribute the
job to ALL matching queues, rather than a single arbitrary queue.

Some terrible alternative approaches I have so far:
1. Have producers submit to a single queue, and have a home-grown "load
balancer" consumer on the other end which distributes jobs to appropriate
queues.
2. When a worker pool spins up, rather than have each worker consume a
dedicate queue, have at least some of them hop between queues in an attempt
to keep them busy.
3. Hacks with timestamps/modulo arithmetic in the routing key to trick the
exchange into being round robin.

Being new to RabbitMQ, my intuition tells me #1 will be a big performance
problem, #2 will be a performance and maintenance problem, and #3 just won't
end up working.  I suspect I'm missing something big here :)

Has anyone successfully load-balanced their workers in a situation like
this?

Any advice much appreciated.
Adam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111010/031ba60c/attachment.htm>