[rabbitmq-discuss] Millions of Queues

[Martin Sustrik]

Ryan,

>>> Also, regarding the 20/80 million queues/bindings, would it be 
>>> possible to partition these s.t. rather than having a single RabbitMQ 
>>> cluster with that number of queues/bindings you could have n 
>>> individual RabbitMQ servers, each with 1/nth of the queues/bindings? 
>>> That might involve your producers having to publish messages to more 
>>> than one broker, and consumers consuming from more than one broker, 
>>> but depending on the exact nature of your app that may not be too 
>>> arduous.
>>>     
>> Yes, we could do this, but it kind of defeats the point of having a 
>> cluster if we have to partition very much manually.  To be more clear 
>> about our use case, we want to create a chat room system, where each 
>> user is a member of K rooms.  The way we're thinking of modeling this is 
>> that each room is represented as a (direct) routing key, and each user 
>> has an individual queue which is then bound to K routing keys.  There's 
>> not really any pattern between users and rooms, so if we partitioned the 
>> queues (and thus users) among clusters, any message sent to an 
>> individual room would have to get sent to every cluster on the (small) 
>> chance that a listening user was in that cluster.  It'd work for a 
>> while, but it can't possibly scale indefinitely.  Is there perhaps a 
>> better way to structure this application?
>>   
> 
> Answering my own question, another way to structure this would be to 
> partition on routing key, such that when sending to or binding to a 
> particular routing key, the client connects to the same cluster.  The 
> downside of this is that, at the extreme, each chat room participant 
> will have to maintain as many connections as rooms they are in (since at 
> the extreme each chat room would be hosted in its own cluster).  But 
> assuming K (the number of chat rooms a user can be in) is finite, then 
> it's not as bad as replicating every message across the entire cluster.

This is a classic example of application that requires pub/sub 
federation. I don't believe it's implemented in RabbitMQ yet, but it 
should be doable.

The idea is that one broker can subscribes for messages to another 
broker, passing any binds/unbinds to the parent broker.

That way you are able to build a tree hierarchy of brokers in your 
datacenter. Thus, root broker would speak to 10 subordinated brokers and 
each of them to 10 more subordinated brokers in its turn. Level 3 of the 
federation would have 100 brokers sharing the queues fairly (80M queues 
/ 100 = 80,000 queues which seems viable). If user subscribes for 
particular topic (bind (myqueue, chatroom)) the bind gets propagated to 
from level 3 broker to level 2 broker and finally to the root broker. At 
that point, incoming messages are routed from root broker to level 2 
broker, then to level 3 broker (where the queue resides) and finally to 
the user.

As for memory requirements, say there are 80M users each being 
subscribed in average to 4 chatrooms. Total number of chatrooms is K. K 
is much less than 80M. On each level 3 broker, there are 80,000 queues, 
so there are 320,000 bindings. On level 2 broker there are 10 
subordinated brokers meaning only 10 queues. Thus, the number of 
bindings will be limited by K - at most K*10 bindings. Same applies to 
the root broker.

No idea whether this helps.
Martin