[rabbitmq-discuss] Millions of Queues
sustrik at imatix.com
Wed Feb 18 06:03:29 GMT 2009
>>> Also, regarding the 20/80 million queues/bindings, would it be
>>> possible to partition these s.t. rather than having a single RabbitMQ
>>> cluster with that number of queues/bindings you could have n
>>> individual RabbitMQ servers, each with 1/nth of the queues/bindings?
>>> That might involve your producers having to publish messages to more
>>> than one broker, and consumers consuming from more than one broker,
>>> but depending on the exact nature of your app that may not be too
>> Yes, we could do this, but it kind of defeats the point of having a
>> cluster if we have to partition very much manually. To be more clear
>> about our use case, we want to create a chat room system, where each
>> user is a member of K rooms. The way we're thinking of modeling this is
>> that each room is represented as a (direct) routing key, and each user
>> has an individual queue which is then bound to K routing keys. There's
>> not really any pattern between users and rooms, so if we partitioned the
>> queues (and thus users) among clusters, any message sent to an
>> individual room would have to get sent to every cluster on the (small)
>> chance that a listening user was in that cluster. It'd work for a
>> while, but it can't possibly scale indefinitely. Is there perhaps a
>> better way to structure this application?
> Answering my own question, another way to structure this would be to
> partition on routing key, such that when sending to or binding to a
> particular routing key, the client connects to the same cluster. The
> downside of this is that, at the extreme, each chat room participant
> will have to maintain as many connections as rooms they are in (since at
> the extreme each chat room would be hosted in its own cluster). But
> assuming K (the number of chat rooms a user can be in) is finite, then
> it's not as bad as replicating every message across the entire cluster.
This is a classic example of application that requires pub/sub
federation. I don't believe it's implemented in RabbitMQ yet, but it
should be doable.
The idea is that one broker can subscribes for messages to another
broker, passing any binds/unbinds to the parent broker.
That way you are able to build a tree hierarchy of brokers in your
datacenter. Thus, root broker would speak to 10 subordinated brokers and
each of them to 10 more subordinated brokers in its turn. Level 3 of the
federation would have 100 brokers sharing the queues fairly (80M queues
/ 100 = 80,000 queues which seems viable). If user subscribes for
particular topic (bind (myqueue, chatroom)) the bind gets propagated to
from level 3 broker to level 2 broker and finally to the root broker. At
that point, incoming messages are routed from root broker to level 2
broker, then to level 3 broker (where the queue resides) and finally to
As for memory requirements, say there are 80M users each being
subscribed in average to 4 chatrooms. Total number of chatrooms is K. K
is much less than 80M. On each level 3 broker, there are 80,000 queues,
so there are 320,000 bindings. On level 2 broker there are 10
subordinated brokers meaning only 10 queues. Thus, the number of
bindings will be limited by K - at most K*10 bindings. Same applies to
the root broker.
No idea whether this helps.
More information about the rabbitmq-discuss