[rabbitmq-discuss] Node scheduling with RabbitMQ

Sat Aug 7 09:43:56 BST 2010

On Aug 6, 2010, at 7:50 PM, Pete Hunt wrote:

> Hi -
> 
> I am currently evaluating RabbitMQ for a master/worker work queue
> environment (like a lot of people do!).
> 
> The first catch with our system is that nodes may fail at any time;
> pretty standard.
> The second catch with our system is that we want the scheduler to be
> cache-aware, that is, since we know what data the task will use ahead
> of time, schedule the task on a node which will result in the least
> cache misses. We already have a system and algorithms in place to
> determine this, however, they need to be integrated with RabbitMQ.
> Would the proper course of action be implementing a custom exchange?
> Or is this outside the scope of RabbitMQ? Please advise; thanks!

cache-aware? You mean data locality?

I've been contemplating this as well.
I think this could be solved by having a proxy in front of RabbitMQ, that routes
the messages to the correct hosts (using routing keys).

E.g. if each worker is also consuming from a private queue, this proxy could
just reroute the message so the task is received by a single worker.

-- 
{Ask Solem,
 Developer ~ Webteam,
 Opera Software,
 +47 98435213 | twitter.com/asksol }.