[rabbitmq-discuss] Single vs. Multiple Queue Topology Performance

[Jerry Kuch]

Hi, Ilya...

> Thanks for your very detailed answer. Regarding availability and
> durability, I definitely plan to use the active/active mirror queue
> approach on the queue. We hope to have the broker be in a "good"
> steady state, in that we'll have enough consumers to empty the queue
> at hopefully the same rate as they come in. The queue is mostly there
> so that we can upgrade the consumers without losing any of the tasks,
> and in case there is a strong spike in traffic, Rabbit can buffer the
> messages for a few seconds.

Rabbit will likely be happier sweating under a burden from consumers
that have failed or become overwhelmed themselves for much longer than
a seconds, subject to available memory, then disk space, then CPU
cycles on the broker system, so your design as proposed should be in
good shape in this respect.

> I have a follow up question if you don't mind:

Absolutely! :-)

> You said "it shouldn't be a bottleneck unless your production rate
> really overwhelms what a single queue process can handle". If there a
> process per queue, aren't we losing the advantages of multiple cores
> helping in the dequeuing to consumers? Wouldn't that be the advantage
> of having multiple queues?

Apologies... my language was imprecise... By "single 'process' per
queue" I meant "single *Erlang process*," not "single full-on, heavy-weight
OS process."

Each queue in Rabbit is implemented as its own Erlang process.  An
Erlang process is really much more like a very lightweight, "green
thread," than it is the OS process that I probably sounded like I was
referring to.

Indeed an "Erlang process" is much lighter weight than an OS thread
or JVM thread (which is backed by an OS thread on all the widely
used JVM implementations today anyway).  Ericsson's choice of the word
'process' in this context is probably unfortunate.  My apologies if I
raised alarm bells by using the word without carefully drawing the
distinction.

To give you an idea of how much more lightweight an Erlang process is
than an OS or Java thread, Erlang applications that have hundreds of
thousands or more "Erlang processes" running within them are not at
all outlandish or exotic in their scale today.  A contemporary JVM
running with half a million java.lang.Thread's alive in it is another
story (memory requirements alone, counting the managed and native stacks,
may have sent you packing as you allocated them in the first place, on
all but the most monstrously powerful hardware).

> The point about premature optimization is well taken, I think it would
> be ideal to have a single queue. Just trying to figure out if that
> puts us at a massive disadvantage since this queue could potentially
> get big at times. Thanks very much!

The size of the queue in terms of the number of messages waiting on it
will ultimately be determined by how much disk space there is to back
the queue if Rabbit comes under memory pressure due to un-ACK-ed
messages and has to start using its internal paging mechanism to move
them to disk and free up working memory (see our documentation on
"memory-based flow control" for details if you're curious).

If you run into scaling limits with a single queue they're likely to
arise from there only being so many cycles/sec available on the
core(s) the queue process is scheduled on before anything else.  If
you really did have straight line throughput hammering a single queue
that so overwhelmed the lifting capacity of a single core, then you'd
want to consider throwing more queues at the process and spreading
load across them since then you could rely on Erlang's scheduler to
spread your work across more physical cores.  Building out your
application starting with a simple queueing topology, and then load
testing to see if you hit that scenario, will tell you when or whether
you need to get more exotic in your design.

Best regards,
Jerry