[rabbitmq-discuss] High-performance routing strategies
matthew at rabbitmq.com
Mon Mar 21 15:43:00 GMT 2011
On Mon, Mar 21, 2011 at 08:35:47AM -0600, Helena Edelson wrote:
> I am curious to know what routing strategies and broker topologies are being
> implemented for performance in similar scenarios of:
> * One broker must handle the load of 1000 distributed agents/clients
> * A cluster of brokers handles 10,000 or more distributed agents/clients
Neither of these scenarios will necessarily induce high load in
100,000 queues on a single broker is fine - I've just created that on my
desktop with no problems. You could likely get to well over a million
with a bit of tuning and enough RAM.
Exchanges and bindings are even cheaper in terms of the memory cost.
Doing 10,000+ connections to a single node is also fine. There is
something somewhere which affects connection creation in RabbitMQ which
causes a log_2(N) slow down on connection establishment: i.e. whilst
initially, I can do about 1000 new connections a second, beyond 10,000
or so connections, the rate of connection establishment slows a little.
What does induce load is binding/exchange churn rate and message rate
and message size.
A sustained load of N thousand binding creation/deletion events per
second will likely overwhelm a single machine, and due to the fact that
such changes have to be distributed to all nodes in a cluster will
likely also induce substantial load on a large cluster too. Isolating
such things really requires stuff like federation which AMQP isn't
really well suited to, in the general case.
More information about the rabbitmq-discuss