[rabbitmq-discuss] Round-robin exchanges?

Tue Oct 11 07:05:12 BST 2011

Adam

Would this solve your problem?

http://twitter.com/#!/hylomorphism/status/117667626880221184

Apologies if I misunderstood.

a

On Tue, Oct 11, 2011 at 2:48 AM, Adam Rabung <adamrabung at gmail.com> wrote:
> Hello,
> I am trying to implement a work queue in RabbitMQ.  I have several machines
> that will act as worker pools for consuming my queues.  I would like each
> job to be processed only once.  Some jobs have environmental requirements,
> ie must be executed on a machine with an SSD.  Not all worker pools will
> meet these requirements.  A first approach would be to have a queue for
> every permutation of requirements: "No Requirements", "Requires SSD",
> "Requires Certificate", etc and have each worker pool subscribe to all
> queues which it can handle.
> A majority of the jobs will have no requirements, so many worker pools will
> be underutilized.  To fight this, I am hoping to distribute jobs in a
> round-robin fashion to all queues that can handle them.  For example, a
> no-requirements job can be handled by any worker pool, and therefore should
> be distributed equally to all of them.  That is, jobs w/ no requirements
> will be evenly distributed to "No Requirements", "Requires SSD", "Requires
> Certificate", etc.
> I tried this with topic exchanges, but of course they will distribute the
> job to ALL matching queues, rather than a single arbitrary queue.
> Some terrible alternative approaches I have so far:
> 1. Have producers submit to a single queue, and have a home-grown "load
> balancer" consumer on the other end which distributes jobs to appropriate
> queues.
> 2. When a worker pool spins up, rather than have each worker consume a
> dedicate queue, have at least some of them hop between queues in an attempt
> to keep them busy.
> 3. Hacks with timestamps/modulo arithmetic in the routing key to trick the
> exchange into being round robin.
> Being new to RabbitMQ, my intuition tells me #1 will be a big performance
> problem, #2 will be a performance and maintenance problem, and #3 just won't
> end up working.  I suspect I'm missing something big here :)
> Has anyone successfully load-balanced their workers in a situation like
> this?
> Any advice much appreciated.
> Adam
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>