[rabbitmq-discuss] Using RabbitMQ for desktop-style messaging in a web app

Thu Dec 8 12:11:29 GMT 2011

On 6 Dec 2011, at 05:04, Bill Moseley wrote:

> We have a lot of background processing that happens in a web app and investigating ways to message users when a background task is complete.  I'm wondering if anyone is using RabbitMQ for something like this and can describe their architecture.
> 
> We might have 20K concurrent users "logged in" at any given time.  Some background tasks are important and the user must be notified, and for those we use the database.  But, there's a number of other tasks that user can trigger where it's not critical that the user is notified, but would be nice to see.  These would be very similar to "you have new mail" in a mail client, for example.

As I see it you have two major options,

1) Use reply-to queues and have the clients browser be a consumer,
using something like amqp-js, Comet, websockets or polling.

  This would be rather experimental, hic sunt dracones.

2) Have the background tasks write notifications to the database,
   which the browsers poll using XMLHttpRequests (or Comet/websockets).

Reply queues may not work well in a web context: the browsers/backend
must be actively consuming to receive the messages, and since the web is stateless
and connections are unreliable it may only be suitable if losing a notification
here and then is acceptable.

You can do like Celery's "amqp result backend" and use a unique id for every
background task, and declare one "result" queue for every task: once when
you publish the task, and then later when you start consuming the result.
That way the browsers query the results using the id, and the messages
will be delivered.  The problem with this approach is that you can end
up with a lot of queues, which can be remedied somewhat by using the x-expires
argument to queue_declare.  (also if you have multiple processes
waiting for the same result, it will only be delivered once, if this is a
problem you can republish the result once you receive it, though this may be prone
to race conditions).

I would go for option 2.

> 
> Questions that have come up is should message queues be session based or user based -- that is if you log into the web app with two browsers (or on two machines) would messages be broadcasted to all sessions for a give user or be session-specific, and how to deal with the non-persistent connections in web apps.
> 
>  Anyone using RabbitMQ for this type of application?
> 