[rabbitmq-discuss] RabbitMQ performance tips, collected with sweat and blood

[Matthew Sackman]

Hi Eugene,

On Fri, Aug 12, 2011 at 08:20:15AM -0700, Eugene Kirpichov wrote:
> I'm surprised to hear about the negative effect; in my experiments the
> positive effect was very large - basically it raised a local stress
> test throughput from several thousand to several hundred thousand
> messages per second; can you please clarify?

With one or a low number of consumers from the same queue, especially if
those consumers are doing very little work and are able to acknowledge
msgs very quickly, if you turn on basic.qos then you'll see that the
loading of the consumers should be far more equal, but overall
throughput will drop. This is because of the additional checks necessary
that a queue has to make before trying to send a message to a queue
(essentially, if the prefetch limit was per consumer, rather than per
channel, life would be a lot simpler, at least from the broker's pov).

If you don't use basic.qos and the consumers are still very fast and
very simple then whilst you might get very unbalanced distribution of
messages to consumers, the overall throughput can be quite a lot higher.
Testing with consumers on the same machine as the broker should
demonstrate this.

But if your consumers are doing quite a lot of processing of each
message and are not able to keep up with the rate of messages that the
broker is sending them then it's quite possible for the broker to flood
one or a few consumers with all the messages and those consumers then
slowly work through their backlog whilst other consumers are left idle:
thus overall throughput can drop. Not only this but the memory
consumption of such consumers may well go through the roof (depending on
the client library being used) as they buffer up all the messages.

So by using basic.qos, you help to ensure that all consumers are kept
equally busy, and you can limit the amount of buffering they each do.
But nevertheless, because the broker has to do more work to achieve this
balancing, the broker spends more CPU per message per consumer, and thus
the broker can be seen to be somewhat slower. Indeed I've measured drops
from over 17kHz to 3kHz by using a single consumer and putting basic.qos
at 1. With basic.qos at some larger values (> 1000), the effect is less
pronounced, but it's still a performance drop for really simple and fast
consumers.

Matthew