[rabbitmq-discuss] [Minimum Air Induction] Introducing Shovel: An AMQP Relay

Mon Sep 22 20:54:18 BST 2008

Valentino,

On Sun, Sep 21, 2008 at 7:58 PM, Ben Hood <0x6e6562 at gmail.com> wrote:
> On Sun, Sep 21, 2008 at 7:26 PM, Matthias Radestock <matthias at lshift.net> wrote:
>>> No, because the current queue delivery mechanism takes a
>>> robin-robin-ish approach, so the overall throughput is rate limited by
>>> the slowest consumer subscribed to a queue.
>>
>> Not true. When messages due for delivery to a particular consumer back up at
>> the TCP/IP level the broker will skip that consumer in its round-robin
>> delivery.
>
> Sure, but only when you hit the limit on a per-queue basis. So, yes,
> you could load balance by saturating the whole thing :-)

After speaking with Matthias today it transpired that he had looked at
a different angle of your question than I did.

Specifically, if you add more consumers to a queue, you will increase
your processing capacity and also be able to scale by doing so.

This will work fine if the consumers are relatively homogeneous wrt
the processing time per job. Under normal circumstances, identical
code should execute in around about the same amount of time on an
identically spec'ed machine, if the respective scheduler allows each
worker to execute in an unthrottled fashion. However when the
processing time becomes heterogeneous across consumers, what you see
is that the overall rate of consumption will be bound by the slowest
consumer, even if processing resources are actually available to the
other workers.

In this situation, the workers are not fully utilizing the available CPU cycles.

The source of this heterogeneity could be resource contention on
specific objects with the execution scope of the worker or something
like the fact that tasks may themselves be heterogeneous in the amount
of resources that are required to complete them.

So the point that Matthias made about the current load balancing is
entirely correct, if a consumer doesn't drain their TCP receive
buffer, they will be skipped.

But this is a coincidental load balancing strategy and only really
works when you have back pressure over the network (which
co-incidentally *may* not kick in if you are using the Erlang clint in
direct mode).

The solution to providing more fine grained control over the flow of
messages across multiple consumers in the QoS feature that I outlined
before. But there is no ETA on this.

HTH,

Ben