[rabbitmq-discuss] Round-robin exchanges?
alexis at rabbitmq.com
Tue Oct 11 07:05:12 BST 2011
Would this solve your problem?
Apologies if I misunderstood.
On Tue, Oct 11, 2011 at 2:48 AM, Adam Rabung <adamrabung at gmail.com> wrote:
> I am trying to implement a work queue in RabbitMQ. I have several machines
> that will act as worker pools for consuming my queues. I would like each
> job to be processed only once. Some jobs have environmental requirements,
> ie must be executed on a machine with an SSD. Not all worker pools will
> meet these requirements. A first approach would be to have a queue for
> every permutation of requirements: "No Requirements", "Requires SSD",
> "Requires Certificate", etc and have each worker pool subscribe to all
> queues which it can handle.
> A majority of the jobs will have no requirements, so many worker pools will
> be underutilized. To fight this, I am hoping to distribute jobs in a
> round-robin fashion to all queues that can handle them. For example, a
> no-requirements job can be handled by any worker pool, and therefore should
> be distributed equally to all of them. That is, jobs w/ no requirements
> will be evenly distributed to "No Requirements", "Requires SSD", "Requires
> Certificate", etc.
> I tried this with topic exchanges, but of course they will distribute the
> job to ALL matching queues, rather than a single arbitrary queue.
> Some terrible alternative approaches I have so far:
> 1. Have producers submit to a single queue, and have a home-grown "load
> balancer" consumer on the other end which distributes jobs to appropriate
> 2. When a worker pool spins up, rather than have each worker consume a
> dedicate queue, have at least some of them hop between queues in an attempt
> to keep them busy.
> 3. Hacks with timestamps/modulo arithmetic in the routing key to trick the
> exchange into being round robin.
> Being new to RabbitMQ, my intuition tells me #1 will be a big performance
> problem, #2 will be a performance and maintenance problem, and #3 just won't
> end up working. I suspect I'm missing something big here :)
> Has anyone successfully load-balanced their workers in a situation like
> Any advice much appreciated.
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
More information about the rabbitmq-discuss