[rabbitmq-discuss] RabbitMQ for Master worker with monitoring?
emile at rabbitmq.com
Thu Dec 23 10:26:23 GMT 2010
On 22/12/10 20:24, yoav glazner wrote:
> On Wed, Dec 22, 2010 at 2:10 PM, Emile Joubert <emile at rabbitmq.com
> <mailto:emile at rabbitmq.com>> wrote:
> Hi Yoav,
> On 21/12/10 21:35, yoav glazner wrote:
> This is my first post to this list, so Hi!
> I'm trying to design a Master/Worker implement-ion with RabbitMQ.
> I have a few requierments due to lagacy issues:
> * works get very long tasks so they should be able to get ABORT
> while working,
> You could send updated status information about work items to a
> dedicated status exchange that workers subscribe to in order to
> obtain abort messages or changes in priority. This assumes that
> tasks are interruptible and that workers have multiple threads of
> that's good
It sounds like you want to be able to communicate two different things
2. task status updates
The updates (abort, reprioritise, pause) to tasks are the difficult part
and the reason why frameworks like Celery won't work. I suggest you
communicate task status updates to workers via a different route from
the way you submit tasks. Translating that to AMQP means using using a
separate exchange and queues, so workers accept tasks via a relatively
slow-moving queue and obtain updates about task status via a different
queue. This assumes that workers have more than one thread of control
for listening on multiple queues and know how to interrupt running
tasks. There may be lots of other ways of accomplishing your goal.
More information about the rabbitmq-discuss