[rabbitmq-discuss] High-performance routing strategies

[Matthew Sackman]

Helena,

On Mon, Mar 21, 2011 at 08:35:47AM -0600, Helena Edelson wrote:
> I am curious to know what routing strategies and broker topologies are being
> implemented for performance in similar scenarios of:
> 
> * One broker must handle the load of 1000 distributed agents/clients
> * A cluster of brokers handles 10,000 or more distributed agents/clients

Neither of these scenarios will necessarily induce high load in
RabbitMQ.

100,000 queues on a single broker is fine - I've just created that on my
desktop with no problems. You could likely get to well over a million
with a bit of tuning and enough RAM.

Exchanges and bindings are even cheaper in terms of the memory cost.

Doing 10,000+ connections to a single node is also fine. There is
something somewhere which affects connection creation in RabbitMQ which
causes a log_2(N) slow down on connection establishment: i.e. whilst
initially, I can do about 1000 new connections a second, beyond 10,000
or so connections, the rate of connection establishment slows a little.

What does induce load is binding/exchange churn rate and message rate
and message size.

A sustained load of N thousand binding creation/deletion events per
second will likely overwhelm a single machine, and due to the fact that
such changes have to be distributed to all nodes in a cluster will
likely also induce substantial load on a large cluster too. Isolating
such things really requires stuff like federation which AMQP isn't
really well suited to, in the general case.

Matthew