[rabbitmq-discuss] Millions of Queues

[Matthias Radestock]

Ryan,

Ryan Williams (Which) wrote:
> Really?  Each node contains a complete copy of the entire mnesia 
> database?

Indeed.

> One thing that we'd 
> really like is for the client to not have too much knowledge of the 
> configuration of the cluster; so it would connect to some well-known 
> host which would then make sure the queues created by the client are 
> made on a host that has spare load.  I imagine that connection.redirect 
> would be helpful here.  What I'm a little mystified about is that the 
> clustering guide implies that rabbit already does this sort of load 
> balancing -- this was why we didn't think it was necessary to run 
> ScalabilityTest on multiple hosts.  But, now that I think about it, 
> ScalabilityTest only opens up one connection for the entire duration of 
> the test so it's not weird that said connection only points at one host.

There is indeed some load balancing going on but, as you say, it will 
only come into play when new connections are established. Also, 
connection.redirect has been removed from version 0-9-1 of the AMQP spec 
- ordinary load balancers should do a much better job so there is no 
point in AMQP attempting to duplicate their functionality.

> another way to structure this would be to partition on routing key,
> such that when sending to or binding to a particular routing key, the
> client connects to the same cluster.  The downside of this is that,
> at the extreme, each chat room participant will have to maintain as
> many connections as rooms they are in (since at the extreme each chat
> room would be hosted in its own cluster).  But assuming K (the number
> of chat rooms a user can be in) is finite, then it's not as bad as
> replicating every message across the entire cluster.

That sounds like a good way to do the partitioning - it requires very 
little knowledge on part of the clients and should scale very well.


Matthias.