[rabbitmq-discuss] RabbitMQ performance tips, collected with sweat and blood

[Eugene Kirpichov]

Hi Matthew,

First, I clearly made a typo when saying "hundred thousand" - it was
tens of thousands (~3000 -> 30000).

Second, I fully agree with what you said. Special measures need to be
taken to avoid causing some consumers to starve.
I tried doing this on application level by setting a value of qos > 1,
but having a buffer that would reject a message that the application
didn't request within a few seconds. On my local tests that worked and
gave large benefit without the problem of starvation; on large-scale
tests it didn't work well - but it was clearly due to an
implementation error of mine somewhere that I didn't have time to
debug.

On Mon, Aug 15, 2011 at 6:04 AM, Matthew Sackman <matthew at rabbitmq.com> wrote:
> Hi Eugene,
>
> On Fri, Aug 12, 2011 at 08:20:15AM -0700, Eugene Kirpichov wrote:
>> I'm surprised to hear about the negative effect; in my experiments the
>> positive effect was very large - basically it raised a local stress
>> test throughput from several thousand to several hundred thousand
>> messages per second; can you please clarify?
>
> With one or a low number of consumers from the same queue, especially if
> those consumers are doing very little work and are able to acknowledge
> msgs very quickly, if you turn on basic.qos then you'll see that the
> loading of the consumers should be far more equal, but overall
> throughput will drop. This is because of the additional checks necessary
> that a queue has to make before trying to send a message to a queue
> (essentially, if the prefetch limit was per consumer, rather than per
> channel, life would be a lot simpler, at least from the broker's pov).
>
> If you don't use basic.qos and the consumers are still very fast and
> very simple then whilst you might get very unbalanced distribution of
> messages to consumers, the overall throughput can be quite a lot higher.
> Testing with consumers on the same machine as the broker should
> demonstrate this.
>
> But if your consumers are doing quite a lot of processing of each
> message and are not able to keep up with the rate of messages that the
> broker is sending them then it's quite possible for the broker to flood
> one or a few consumers with all the messages and those consumers then
> slowly work through their backlog whilst other consumers are left idle:
> thus overall throughput can drop. Not only this but the memory
> consumption of such consumers may well go through the roof (depending on
> the client library being used) as they buffer up all the messages.
>
> So by using basic.qos, you help to ensure that all consumers are kept
> equally busy, and you can limit the amount of buffering they each do.
> But nevertheless, because the broker has to do more work to achieve this
> balancing, the broker spends more CPU per message per consumer, and thus
> the broker can be seen to be somewhat slower. Indeed I've measured drops
> from over 17kHz to 3kHz by using a single consumer and putting basic.qos
> at 1. With basic.qos at some larger values (> 1000), the effect is less
> pronounced, but it's still a performance drop for really simple and fast
> consumers.
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>



-- 
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/