[rabbitmq-discuss] Single vs. Multiple Queue Topology Performance

[Ilya Volodarsky]

Thanks Jerry! Your answers were very thorough and helpful. I'll build
it with one queue, and compare it to the other alternative if the load
testing shows its necessary. Thanks again!

On Nov 4, 3:22 pm, Jerry Kuch <jer... at vmware.com> wrote:
> Hi, Ilya...
>
> > Thanks for your very detailed answer. Regarding availability and
> > durability, I definitely plan to use the active/active mirror queue
> > approach on the queue. We hope to have the broker be in a "good"
> > steady state, in that we'll have enough consumers to empty the queue
> > at hopefully the same rate as they come in. The queue is mostly there
> > so that we can upgrade the consumers without losing any of the tasks,
> > and in case there is a strong spike in traffic, Rabbit can buffer the
> > messages for a few seconds.
>
> Rabbit will likely be happier sweating under a burden from consumers
> that have failed or become overwhelmed themselves for much longer than
> a seconds, subject to available memory, then disk space, then CPU
> cycles on the broker system, so your design as proposed should be in
> good shape in this respect.
>
> > I have a follow up question if you don't mind:
>
> Absolutely! :-)
>
> > You said "it shouldn't be a bottleneck unless your production rate
> > really overwhelms what a single queue process can handle". If there a
> > process per queue, aren't we losing the advantages of multiple cores
> > helping in the dequeuing to consumers? Wouldn't that be the advantage
> > of having multiple queues?
>
> Apologies... my language was imprecise... By "single 'process' per
> queue" I meant "single *Erlang process*," not "single full-on, heavy-weight
> OS process."
>
> Each queue in Rabbit is implemented as its own Erlang process.  An
> Erlang process is really much more like a very lightweight, "green
> thread," than it is the OS process that I probably sounded like I was
> referring to.
>
> Indeed an "Erlang process" is much lighter weight than an OS thread
> or JVM thread (which is backed by an OS thread on all the widely
> used JVM implementations today anyway).  Ericsson's choice of the word
> 'process' in this context is probably unfortunate.  My apologies if I
> raised alarm bells by using the word without carefully drawing the
> distinction.
>
> To give you an idea of how much more lightweight an Erlang process is
> than an OS or Java thread, Erlang applications that have hundreds of
> thousands or more "Erlang processes" running within them are not at
> all outlandish or exotic in their scale today.  A contemporary JVM
> running with half a million java.lang.Thread's alive in it is another
> story (memory requirements alone, counting the managed and native stacks,
> may have sent you packing as you allocated them in the first place, on
> all but the most monstrously powerful hardware).
>
> > The point about premature optimization is well taken, I think it would
> > be ideal to have a single queue. Just trying to figure out if that
> > puts us at a massive disadvantage since this queue could potentially
> > get big at times. Thanks very much!
>
> The size of the queue in terms of the number of messages waiting on it
> will ultimately be determined by how much disk space there is to back
> the queue if Rabbit comes under memory pressure due to un-ACK-ed
> messages and has to start using its internal paging mechanism to move
> them to disk and free up working memory (see our documentation on
> "memory-based flow control" for details if you're curious).
>
> If you run into scaling limits with a single queue they're likely to
> arise from there only being so many cycles/sec available on the
> core(s) the queue process is scheduled on before anything else.  If
> you really did have straight line throughput hammering a single queue
> that so overwhelmed the lifting capacity of a single core, then you'd
> want to consider throwing more queues at the process and spreading
> load across them since then you could rely on Erlang's scheduler to
> spread your work across more physical cores.  Building out your
> application starting with a simple queueing topology, and then load
> testing to see if you hit that scenario, will tell you when or whether
> you need to get more exotic in your design.
>
> Best regards,
> Jerry
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss