[rabbitmq-discuss] RabbitMQ for Master worker with monitoring?

Wed Dec 22 20:50:00 GMT 2010

Have you looked into celery? http://celeryproject.org/. You might be able to bolt the additional features you need on top or use it as-is with creative segmenting / driver scripts.

On Dec 22, 2010, at 12:24 PM, yoav glazner wrote:

> 
> On Wed, Dec 22, 2010 at 2:10 PM, Emile Joubert <emile at rabbitmq.com> wrote:
> Hi Yoav,
> 
> 
> On 21/12/10 21:35, yoav glazner wrote:
> This is my first post to this list, so Hi!
> I'm trying to design a Master/Worker implement-ion with RabbitMQ.
> I have a few requierments due to lagacy issues:
> * works get very long tasks so they should be able to get ABORT messages
> while working,
> 
> You could send updated status information about work items to a dedicated status exchange that workers subscribe to in order to obtain abort messages or changes in priority. This assumes that tasks are interruptible and that workers have multiple threads of control.
> that's good 
> 
> 
> * no task can be lost
> 
> The use persistence and transactions is indicated.\
> yes, I saw that. 
>  
> 
> * each task has priority and that priority may change when the task is
> already in a queue.
> 
> If the task is stuck in a long queue you might be able to send a cancel message via the status exchange and requeue the task at the correct priority.
>  
> I have no idea what this means :( 
> 
> 
> * tasks can be removed from the work queue (user wants to pause/cancel
> an operation)
> 
> Depending on exactly what pausing means, this could be accomplished to sending an appropriate message to the status exchange.
> again, no clue on this, what is the "status exchange"? 
> 
> 
> * An client of rabbitMQ can fetch the Pending tasks to know what is the
> status of the big operation.
> 
> If you explain what a bit more about your problem I might be able to make a suggestion here - what is a big operation?
> 
> This big operation is a crypto-calculation on many number, the operation is devided to many tasks (each can take a few hours), so I want to be able to know what is the operation status. (I can have a few big operations in the same time)
> 
> After googling I found out that for priorities I'll need a few work
> queues, but will I be able to switch a priority of a task in one
> transaction?
> 
> No.
> ok, but You said:
> "If the task is stuck in a long queue you might be able to send a cancel message via the status exchange and requeue the task at the correct priority." 
> so I guess this will be transaction-less?
> 
> 
> 
> Regards
> 
> Emile
> 
> Thanks!
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20101222/db677851/attachment.htm>