[rabbitmq-discuss] Millions of Queues

[Ryan Williams (Which)]

Ryan Williams (Which) wrote:
>> Also, regarding the 20/80 million queues/bindings, would it be 
>> possible to partition these s.t. rather than having a single RabbitMQ 
>> cluster with that number of queues/bindings you could have n 
>> individual RabbitMQ servers, each with 1/nth of the queues/bindings? 
>> That might involve your producers having to publish messages to more 
>> than one broker, and consumers consuming from more than one broker, 
>> but depending on the exact nature of your app that may not be too 
>> arduous.
>>     
> Yes, we could do this, but it kind of defeats the point of having a 
> cluster if we have to partition very much manually.  To be more clear 
> about our use case, we want to create a chat room system, where each 
> user is a member of K rooms.  The way we're thinking of modeling this is 
> that each room is represented as a (direct) routing key, and each user 
> has an individual queue which is then bound to K routing keys.  There's 
> not really any pattern between users and rooms, so if we partitioned the 
> queues (and thus users) among clusters, any message sent to an 
> individual room would have to get sent to every cluster on the (small) 
> chance that a listening user was in that cluster.  It'd work for a 
> while, but it can't possibly scale indefinitely.  Is there perhaps a 
> better way to structure this application?
>   

Answering my own question, another way to structure this would be to 
partition on routing key, such that when sending to or binding to a 
particular routing key, the client connects to the same cluster.  The 
downside of this is that, at the extreme, each chat room participant 
will have to maintain as many connections as rooms they are in (since at 
the extreme each chat room would be hosted in its own cluster).  But 
assuming K (the number of chat rooms a user can be in) is finite, then 
it's not as bad as replicating every message across the entire cluster.