[rabbitmq-discuss] Load balancing with multiple consumers on a single queue

Wed Jul 30 19:08:25 BST 2008

I just noticed David is using Java, so my Erlang-related comments won't be
helpful. Timers and processes have much lower overhead in Erlang than in
other languages so what works for Erlang probably won't work as well in
these languages. I also noticed that things ground to a halt when using 500
consumers doing basic.get in a polling fashion. I'd be very curious to know
what polling interval was used to get RabbitMQ to grind to a halt with 500
consumers.

On Wed, Jul 30, 2008 at 1:52 PM, Edwin Fine
<rabbitmq-discuss_efine at usa.net>wrote:

> I think I was the one with the basic.qos problem because in stress testing,
> my consumers (started with basic.consume) were being flooded with messages
> and could not keep up, so their Erlang message queues became huge and
> eventually there were memory problems. What I eventually wound up doing was,
> indeed, changing the code to use basic.get with a short timeout. This keeps
> the messages in RabbitMQ's camp. Since they are persistent I imagine they
> will be stored on disk and not take too much memory. Essentially, when I
> receive a basic.get_empty, I call erlang.send_after() to kick off a new
> basic.get in a short while. When the queues are busy, there will be no
> overhead; it's only when idle that there will be a number of timers
> expiring.
>
> So far I have not seen significant performance hits, but then again, I am
> only using 50 queues and and aggregate of 140 messages/second. Maybe
> Matthias's suggestion will work for you.
>
> Hope this helps.
>
> Edwin
>
>
> On Wed, Jul 30, 2008 at 1:26 PM, Matthias Radestock <matthias at lshift.net>wrote:
>
>> David.Corcoran at edftrading.com wrote:
>> > I'm not sure how message delivery credits work but having read through a
>> > little it looks like it might be size oriented.
>>
>> basic.qos can do count based windows too. So with a window size of one
>> you can guarantee that at most one message is "in flight" or being
>> processed by the consumer.
>>
>> This has come up before, and basic.qos is on our todo list. As with
>> basic.reject, and perhaps more so, there are quite a few challenges in
>> implementing it, which is why it hasn't been done yet. One difficulty is
>> that basic.qos operates on an entire channel (or, optionally, even an
>> entire connection). Since a single channel can consume messages from
>> multiple queues, and these queues can have consumers on other channels
>> (with consumption possibly limited by basic.qos) we basically have a
>> nice little optimisation & consensus problem at our hands: figuring out
>> which queues should send which messages to which channels such that the
>> maximum number of messages gets delivered while staying inside the
>> configured qos limits. And all that in a distributed setting. And the
>> limits change all the time outside the brokers control - clients ack
>> messages whenever they please, the topology may change, etc, etc. Great
>> fun :)
>>
>>
>> Matthias.
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>
>
> --
> For every expert there is an equal and opposite expert - Arthur C. Clarke
>

-- 
For every expert there is an equal and opposite expert - Arthur C. Clarke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20080730/990d848d/attachment.htm 