[rabbitmq-discuss] Newbee consulting question

[Tomer Paz]

Hi.

I am examining RabbitMQ as an AMQP broker for a workflow system that we 
wish to develop in-house, on-premise.
The closest description to our architecture is Amazon SWF. it is ~95% the 
solution concept to our requirements, but we can't use it as it is a cloud 
SaaS and our requirement is 'on-premise' :)

I was wondering How to manage distributed Tasks where the task manager 
('orchestrator') is at the data-center, 
 the workers are distributed in different physical sites (some on the same 
site, some far in other site separated by WAN),
 where the connection between the datacenter and the sites might be 
disrupted sometimes.

We don't have a major performance requirement, we need not thousands of 
msg/sec, more like 1-3k messages a minute. Each worker might work for 
minutes on a single task...

Since the requirement is for WorkFlow management, it means we want tasks 
marked done by a remote worker, will be reported back to the manager, to 
keep the State of all tasks managed in central manner.
We use that for monitoring and reports (statistics).
I have read about the RPC like pattern, prefer to avoid it (!)
I also read the tutorials about task/worker pattern which is closer to what 
we need, but we also need the capability to parallelise 'steps' (that's 
what workflow orchestration is about) within a parent task so that
some tasks in the workflow tree will be processed consequently, some in 
parallel. 
Assume workers are not necessarily the same. i.e. there are different task 
"types" thus different workers processing them (hence this is not a simple 
pub/sub pattern either).

It sounds like we need some kind of mixture of three patterns to pull this 
out, but I am not sure: task/worker, pub/sub and the "Topic" Exchange kind 
of routing.
Oh, and did I mention we want the architecture to be "pull" rather than 
"push" in terms of workers 'asking' the workflow manager to get new tasks 
when they are free, 
 rather than task manager pushing tasks to workers blindly.

Would love to hear some experience if there is such, with such pattern 
using AMQ and RubbitMQ in particular to communicate the tasks.

TIA
Tomer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130409/25ca3e17/attachment.htm>