[rabbitmq-discuss] single broker and queue scalability question

[David Wragg]

"justin.adam.miller at gmail.com" <justin.adam.miller at gmail.com> writes:
> I have an app that will have, at any one time, < 500 connections to a
> single broker, across multiple machines, all of whom I want as
> consumers of one queue so that rabbit can load balance amongst them.

It's certainly possible for rabbit to handle 500 connections, even on
relatively modest hardware.

As for 500 consumers of a single queue, it doesn't seem likely to be a
problem.  But you could prototype it and see what you get.  Try to match
your expected message throughput, message size distribution, and
consumption intervals (how long between a message being delivered and
being acked).  And try with hardware similar or less capable than what
you expect to use in production.

> Is handling so few connections/consumers a joke for rabbit? Should I
> consider clustering, and if so, what does clustering buy me in this
> case? The one thing I don't love about clustering is that, without
> active/passive failover, the node that declares the queue is a single
> point of failure.

Currently clustering won't buy you anything in this context.

As for active/active failover, there will be developments in that area
soon.  But even then, a single queue will remain a potential bottleneck.

> The idea of clustering along with multiple queues bound to one
> exchange is appealing (since exchange and binding information is
> replicated across the cluster), but rabbit will not round robin
> messages to all the bound queues -- there's no exchange type (that I
> know of) that has that behavior.
>
> If I knew the bound queues ahead of time, then I could pick one at
> random, but it seems like determining that queue would be cumbersome,
> inefficient and/or costly at best.

If you were to go down this route, there may be advantages to using
multiple brokers rather than clustering.  E.g., have one queue in each
broker, with each consumer coonsuming from one of them.  Then each
producer connects to each broker and publishes each message to one of
them, chosen at random.  This would allow you to do things like rolling
upgrades of the brokers.  Clustering won't, as we don't support mixed
version clusters.

-- 
David Wragg
Staff Engineer, RabbitMQ
VMware, Inc.