[rabbitmq-discuss] Load balancing with multiple consumers on a single queue

[Edwin Fine]

I was curious about the cost of polling using basic.get, so I wrote a small
Erlang client program to start up 500 consumers, on one TCP/IP channel,
listening on one queue, each polling with a basic.get every 100ms. I know
this is a high rate, but I wanted to see how it behaved.

Maybe my code was naive or inefficient, but I found the polling program's
Linux process to be taking 85% CPU (and off the chart in appmon), while
Rabbit's process was about 30% (on a 4-core Intel Q6600 box, hence CPU
numbers don't add up to 100%). I haven't tried it with the "native" client.
It seemed to make no difference if kernel poll was enabled.

Regards,
Edwin

Environment: Ubuntu Linux 8.04, 2.4 GHz Q6600, 8 GB RAM, Erlang R12B-3,
RabbitMQ 1.4.0, RabbitMQ Erlang client as of 2008/06/23.

On Wed, Jul 30, 2008 at 2:08 PM, Edwin Fine
<rabbitmq-discuss_efine at usa.net>wrote:

> I just noticed David is using Java, so my Erlang-related comments won't be
> helpful. Timers and processes have much lower overhead in Erlang than in
> other languages so what works for Erlang probably won't work as well in
> these languages. I also noticed that things ground to a halt when using 500
> consumers doing basic.get in a polling fashion. I'd be very curious to know
> what polling interval was used to get RabbitMQ to grind to a halt with 500
> consumers.
>
>
> On Wed, Jul 30, 2008 at 1:52 PM, Edwin Fine <
> rabbitmq-discuss_efine at usa.net> wrote:
>
>> I think I was the one with the basic.qos problem because in stress
>> testing, my consumers (started with basic.consume) were being flooded with
>> messages and could not keep up, so their Erlang message queues became huge
>> and eventually there were memory problems. What I eventually wound up doing
>> was, indeed, changing the code to use basic.get with a short timeout. This
>> keeps the messages in RabbitMQ's camp. Since they are persistent I imagine
>> they will be stored on disk and not take too much memory. Essentially, when
>> I receive a basic.get_empty, I call erlang.send_after() to kick off a new
>> basic.get in a short while. When the queues are busy, there will be no
>> overhead; it's only when idle that there will be a number of timers
>> expiring.
>>
>> So far I have not seen significant performance hits, but then again, I am
>> only using 50 queues and and aggregate of 140 messages/second. Maybe
>> Matthias's suggestion will work for you.
>>
>> Hope this helps.
>>
>> Edwin
>>
>>
>> On Wed, Jul 30, 2008 at 1:26 PM, Matthias Radestock <matthias at lshift.net>wrote:
>>
>>> David.Corcoran at edftrading.com wrote:
>>> > I'm not sure how message delivery credits work but having read through
>>> a
>>> > little it looks like it might be size oriented.
>>>
>>> basic.qos can do count based windows too. So with a window size of one
>>> you can guarantee that at most one message is "in flight" or being
>>> processed by the consumer.
>>>
>>> This has come up before, and basic.qos is on our todo list. As with
>>> basic.reject, and perhaps more so, there are quite a few challenges in
>>> implementing it, which is why it hasn't been done yet. One difficulty is
>>> that basic.qos operates on an entire channel (or, optionally, even an
>>> entire connection). Since a single channel can consume messages from
>>> multiple queues, and these queues can have consumers on other channels
>>> (with consumption possibly limited by basic.qos) we basically have a
>>> nice little optimisation & consensus problem at our hands: figuring out
>>> which queues should send which messages to which channels such that the
>>> maximum number of messages gets delivered while staying inside the
>>> configured qos limits. And all that in a distributed setting. And the
>>> limits change all the time outside the brokers control - clients ack
>>> messages whenever they please, the topology may change, etc, etc. Great
>>> fun :)
>>>
>>>
>>> Matthias.
>>>
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>
>>>
>>
>>
>> --
>> For every expert there is an equal and opposite expert - Arthur C. Clarke
>>
>
>
>
> --
> For every expert there is an equal and opposite expert - Arthur C. Clarke
>



-- 
For every expert there is an equal and opposite expert - Arthur C. Clarke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20080730/3ee0d301/attachment.htm