[rabbitmq-discuss] Design for Comet

[Marek Majkowski]

On Thu, Feb 23, 2012 at 16:54, tsuraan <tsuraan at gmail.com> wrote:
> We have a comet-like server that has four basic tiers of messages:
> page-level, user-level, topic, and global.  When a page is rendered
> that hooks into the comet system, it initially subscribes to the comet
> server using a POST with the tiers that it wants, and then it uses GET
> to wait on new messages using long-polling.  To support this, I've
> made a somewhat overly-complicated server that does a lot of internal
> queueing, but is ultimately backed by Rabbit.
>
> I've been looking at simplifying this, but first I thought I'd run it
> past you guys, since I think my idea may have some components that
> rabbit would hate.  Basically, what I want to do is make a short-lived
> (and non-durable) queue per page (where a single page is dynamic, so
> it tends to live for a while), and then have that queue be subscribed
> to a "global" exchange, a "user" exchange, and a "topics" exchange.

Does it mean that every page is user specific? Or is "user" exchange
not for per-user messages but rather global ones?

> I see that Rabbit now has a queue-expiry extension, so unused queues
> could be automatically garbage collected (I was thinking of basically
> having a 10s idle timeout on my temp queues).

I'm not sure what your scale is, but this setup has a good property -
it should be relatively easy to shard it. So it's generally reasonable.

> My concern is whether
> rabbit likes having thousands or tens of thousands of queues created
> and torn down every few minutes.  I seem to recall that it didn't like
> that at all in the past, but Rabbit's been progressing like mad
> lately, so I figured I'd ask if that's a sane thing to do these days.

Like always - you need to prototype and benchmark it :)

But generally, I would probably go for a different setup.

I think we can assume with reasonable confidence that number
of concurrent users on your site is constant. (as opposed to
dynamic pages AFAIU)

So why not create queue-per-user? (for users actively browsing).

When user enters a page - you can subscribe the queue to a per-page
exchange. When user navigates away - you unsubscribe.

Of course, this has some limitations - you may want to buffer
data per-page, and this setup can't do that easily (you would
need to use redis or something for that).

Hope that helps,
  Marek