[rabbitmq-discuss] Load balancing with multiple consumers on a single queue

[Edwin Fine]

I think I was the one with the basic.qos problem because in stress testing,
my consumers (started with basic.consume) were being flooded with messages
and could not keep up, so their Erlang message queues became huge and
eventually there were memory problems. What I eventually wound up doing was,
indeed, changing the code to use basic.get with a short timeout. This keeps
the messages in RabbitMQ's camp. Since they are persistent I imagine they
will be stored on disk and not take too much memory. Essentially, when I
receive a basic.get_empty, I call erlang.send_after() to kick off a new
basic.get in a short while. When the queues are busy, there will be no
overhead; it's only when idle that there will be a number of timers
expiring.

So far I have not seen significant performance hits, but then again, I am
only using 50 queues and and aggregate of 140 messages/second. Maybe
Matthias's suggestion will work for you.

Hope this helps.

Edwin

On Wed, Jul 30, 2008 at 1:26 PM, Matthias Radestock <matthias at lshift.net>wrote:

> David.Corcoran at edftrading.com wrote:
> > I'm not sure how message delivery credits work but having read through a
> > little it looks like it might be size oriented.
>
> basic.qos can do count based windows too. So with a window size of one
> you can guarantee that at most one message is "in flight" or being
> processed by the consumer.
>
> This has come up before, and basic.qos is on our todo list. As with
> basic.reject, and perhaps more so, there are quite a few challenges in
> implementing it, which is why it hasn't been done yet. One difficulty is
> that basic.qos operates on an entire channel (or, optionally, even an
> entire connection). Since a single channel can consume messages from
> multiple queues, and these queues can have consumers on other channels
> (with consumption possibly limited by basic.qos) we basically have a
> nice little optimisation & consensus problem at our hands: figuring out
> which queues should send which messages to which channels such that the
> maximum number of messages gets delivered while staying inside the
> configured qos limits. And all that in a distributed setting. And the
> limits change all the time outside the brokers control - clients ack
> messages whenever they please, the topology may change, etc, etc. Great
> fun :)
>
>
> Matthias.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>


-- 
For every expert there is an equal and opposite expert - Arthur C. Clarke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20080730/46fc6813/attachment.htm