[rabbitmq-discuss] Scaling to large number of bindings

[Aaron Dufour]

I'm having some trouble coming up with a scalable RabbitMQ setup for our 
application (we're currently using Redis pub/sub but that's going to become 
a problem in the near future).  At our current level, we would have ~200 
queues with ~500,000 bindings between them, but we're seeing consistent 
growth and expect the number of bindings to increase quite a bit beyond 
that.  There's no logical partitioning among these queues - any queue can 
expect a message from any producer.

My plan so far consists of having 3 layers of RabbitMQ servers - an 
incoming layer, where each server accepts messages from a portion of our 
producers; a routing layer, where each server routes a portion of the 
messages based on routing key (potentially each routing server accepting 
from each incoming server); and an outgoing layer, where each server will 
contain a portion of the outgoing queues.  The routing layer would contain 
two exchanges per server, linked with an exchange-to-exchange binding where 
the binding key defines which messages that that routing server is dealing 
with, and the incoming exchange is federated to the exchange on each 
incoming server.  It appears that this will operate as I would like and 
deal with increased load well.

On the other end, I would have an exchange on each outgoing server that the 
queues would bind to (a direct exchange - they are binding to exact routing 
keys).  However, when I federate this to the outgoing exchanges on the 
routing servers, every binding is stored on every routing server, which 
puts a hard ceiling on the number of bindings I can make.  Ideally the 
outgoing servers would only tell each routing server about the bindings it 
cares about (since each routing server is only dealing with a known subset 
of messages, based on routing keys), but I can't see a way to make this 
happen.

Do I need something other than federation?  Am I thinking about the whole 
problem wrong?  I could use some help figuring out the right topology to 
allow the system to scale for the foreseeable future.

Thanks,
Aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140527/f21ab006/attachment.html>