[rabbitmq-discuss] RabbitMQ for Master worker with monitoring?

[Emile Joubert]

Hi Yoav

On 22/12/10 20:24, yoav glazner wrote:
>
> On Wed, Dec 22, 2010 at 2:10 PM, Emile Joubert <emile at rabbitmq.com
> <mailto:emile at rabbitmq.com>> wrote:
>
>     Hi Yoav,
>
>
>     On 21/12/10 21:35, yoav glazner wrote:
>
>         This is my first post to this list, so Hi!
>         I'm trying to design a Master/Worker implement-ion with RabbitMQ.
>         I have a few requierments due to lagacy issues:
>         * works get very long tasks so they should be able to get ABORT
>         messages
>         while working,
>
>
>     You could send updated status information about work items to a
>     dedicated status exchange that workers subscribe to in order to
>     obtain abort messages or changes in priority. This assumes that
>     tasks are interruptible and that workers have multiple threads of
>     control.
>
> that's good

It sounds like you want to be able to communicate two different things 
to workers:

1. tasks
2. task status updates

The updates (abort, reprioritise, pause) to tasks are the difficult part 
and the reason why frameworks like Celery won't work. I suggest you 
communicate task status updates to workers via a different route from 
the way you submit tasks. Translating that to AMQP means using using a 
separate exchange and queues, so workers accept tasks via a relatively 
slow-moving queue and obtain updates about task status via a different 
queue. This assumes that workers have more than one thread of control 
for listening on multiple queues and know how to interrupt running 
tasks. There may be lots of other ways of accomplishing your goal.

Regards

Emile