[rabbitmq-discuss] Architecture Question for webservice

Thu Mar 25 17:27:21 GMT 2010

On Tue, Mar 23, 2010 at 1:31 PM, Jonathan Stott <jonathan.stott at gmail.com>wrote:

> Hi all
>
> I'm setting up a simple webservice.  Since it involves quite a few file
> operations and some potentially long running tasks, I want those tasks
> handled by a separate process (or potentially, processes).  This also lets
> me separate concerns.  So I want to use rabbitmq to communicate between the
> two (and handle the mechanics of queueing jobs).
>
> The job submission side is easy.  I route messages to a queue the worker(s)
> is subscribed to.  It pulls the jobs off the queue, does the work.  But what
> if I want to send back a confirmation?  The obvious way is to set up a queue
> to route back messages to the front end process.  I can even send a 'return
> to' 'address' along with the job.
>
> My question is:  How many queues do I need to send back to?
>
> If I have one queue for the return messages, then how do I split out
> different job outputs from the stream? so that I send correct status reports
> to each client.
>
> But if each job has the webservice setup a queue just to listen for the
> reply, I'll be setting up and tearing down queues all the time.
>
> So which way would people recommend or am I thinking about this the wrong
> way?  Is there an option I've missed?  Is there something which would fit
> better than rabbit?
>
> Regards
> Jon
>
>
I use something similar for "load balancing" some scripting requests in my
system. Essentially two queues and one manager that exposes the API and
communicates with the workers over the two queues, one work request queue
and one for replies. The requests get a correlation id and the manager uses
that to find the appropriate client request to return the result to.

Now, this uses Erlang, where the gen_server behaviour copes rather elegantly
with long running server requests using noreply return values from the
initial call and later sending the reply using the gen_server:reply/2
function.

In essence, the initial work call creates a correlation id and stores that
together with the callers process id. When a worker sends a reply, it quotes
the request's correlation id. The manager then uses that to find the pid
that needs to receive the result.

Sorry for the slight vagueness of this, I'm on holiday and it would take me
a little too long to swap the knowledge back into my brain at the moment.

Robby
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100325/bd675656/attachment.htm 