[rabbitmq-discuss] Node scheduling with RabbitMQ
askh at opera.com
Sat Aug 7 09:43:56 BST 2010
On Aug 6, 2010, at 7:50 PM, Pete Hunt wrote:
> Hi -
> I am currently evaluating RabbitMQ for a master/worker work queue
> environment (like a lot of people do!).
> The first catch with our system is that nodes may fail at any time;
> pretty standard.
> The second catch with our system is that we want the scheduler to be
> cache-aware, that is, since we know what data the task will use ahead
> of time, schedule the task on a node which will result in the least
> cache misses. We already have a system and algorithms in place to
> determine this, however, they need to be integrated with RabbitMQ.
> Would the proper course of action be implementing a custom exchange?
> Or is this outside the scope of RabbitMQ? Please advise; thanks!
cache-aware? You mean data locality?
I've been contemplating this as well.
I think this could be solved by having a proxy in front of RabbitMQ, that routes
the messages to the correct hosts (using routing keys).
E.g. if each worker is also consuming from a private queue, this proxy could
just reroute the message so the task is received by a single worker.
Developer ~ Webteam,
+47 98435213 | twitter.com/asksol }.
More information about the rabbitmq-discuss