[rabbitmq-discuss] RabbitMQ performance tips, collected with sweat and blood

[Matthew Sackman]

On Thu, Aug 11, 2011 at 09:47:40PM -0700, Eugene Kirpichov wrote:
> * Turn on prefetch. Large is good, small is better than nothing. You
> can try a trick "reject a prefetched message if we didn't actually
> need it for a few seconds/minutes". Consider prefetch individually for
> each queue; e.g. it may be that you cannot turn it on for tasks, but
> can for results.

I don't know what you mean by prefetch. If you mean basic.qos then
you're a little mistaken - basic.qos is a per channel thing, and not per
consumer. Thus setting basic.qos affects every queue from which you're
consuming within the same channel. Using basic.qos with non-zero values
is known to have a fairly large negative impact on performance.

> * Turn on autoAck

autoAck and basic.qos are mutually exclusive.

> * In a cluster, always communicate with a queue only through the
> broker that first declared it

Why do you say that?

> * Avoid frequently doing these things, esp. on a cluster: establishing
> a connection, BasicConsume, QueueDeclare etc. If you do them often
> (dozens/hundreds of times per second), you're completely screwed. And
> conversely, if you're completely screwed, the first place to look is
> whether you're doing these things too frequently, and if you're not,
> look again.

Connection establishment certainly can be a little expensive but you
should be able to do a few hundred a second fairly easily. Things that
hit queues when those queues are already busy can take some time
depending on the loading present on the queue. This is something we're
aware of and are trying to figure out how to solve.

> * Increase erlang process limit

We do that by default, but certainly for large installations raising
that further may be necessary.

> * Split one queue into many, as a queue can only utilize 1 core -
> across a cluster or not (either will give some benefit); publish in
> roundrobin (between those sub-queues that aren't overloaded acc. to
> publisher confirmations); consume all simultaneously.

Yeah, the problem with that is that your message stream may require some
sort of ordering guarantees and by publishing round-robin, you're going
to lose that. For some use cases though, this can be an effective
approach.

Matthew