[rabbitmq-discuss] High-performance routing strategies

Mon Mar 21 15:46:39 GMT 2011

Hi,

I was testing queue number limits the other day, one thing to watch is the number of erlang processes. If I'm not mistaken a new queue implies a new Erlang Process. Probably the same with exchanges?

Cheers,

Alvaro

On Mar 21, 2011, at 4:43 PM, Matthew Sackman wrote:

> Helena,
> 
> On Mon, Mar 21, 2011 at 08:35:47AM -0600, Helena Edelson wrote:
>> I am curious to know what routing strategies and broker topologies are being
>> implemented for performance in similar scenarios of:
>> 
>> * One broker must handle the load of 1000 distributed agents/clients
>> * A cluster of brokers handles 10,000 or more distributed agents/clients
> 
> Neither of these scenarios will necessarily induce high load in
> RabbitMQ.
> 
> 100,000 queues on a single broker is fine - I've just created that on my
> desktop with no problems. You could likely get to well over a million
> with a bit of tuning and enough RAM.
> 
> Exchanges and bindings are even cheaper in terms of the memory cost.
> 
> Doing 10,000+ connections to a single node is also fine. There is
> something somewhere which affects connection creation in RabbitMQ which
> causes a log_2(N) slow down on connection establishment: i.e. whilst
> initially, I can do about 1000 new connections a second, beyond 10,000
> or so connections, the rate of connection establishment slows a little.
> 
> What does induce load is binding/exchange churn rate and message rate
> and message size.
> 
> A sustained load of N thousand binding creation/deletion events per
> second will likely overwhelm a single machine, and due to the fact that
> such changes have to be distributed to all nodes in a cluster will
> likely also induce substantial load on a large cluster too. Isolating
> such things really requires stuff like federation which AMQP isn't
> really well suited to, in the general case.
> 
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss