[rabbitmq-discuss] RabbitMQ performance tips, collected with sweat and blood

[Eugene Kirpichov]

Hi,

I decided that I've asked too many hard or confusing questions lately,
and decided to contribute instead :)

So here's what I learnt about making rabbitmq fast, or at least
avoiding making it slow:
* Turn on prefetch. Large is good, small is better than nothing. You
can try a trick "reject a prefetched message if we didn't actually
need it for a few seconds/minutes". Consider prefetch individually for
each queue; e.g. it may be that you cannot turn it on for tasks, but
can for results.
* Turn on autoAck
* Do not use transactions
* In a cluster, always communicate with a queue only through the
broker that first declared it
* Avoid frequently doing these things, esp. on a cluster: establishing
a connection, BasicConsume, QueueDeclare etc. If you do them often
(dozens/hundreds of times per second), you're completely screwed. And
conversely, if you're completely screwed, the first place to look is
whether you're doing these things too frequently, and if you're not,
look again.
* Avoid overloading rabbitmq with published messages in a queue and
making it swap like crazy. For that, use ConfirmSelect (publisher
confirmations), track (in your app) the number of unconfirmed messages
and block further submission if there are too many (more than a few
dozen megabytes). The effect is really dramatic.
* Use a 64-bit machine and raise the vm high watermark
* Increase erlang process limit
* Increase file descriptor limit
* Split one queue into many, as a queue can only utilize 1 core -
across a cluster or not (either will give some benefit); publish in
roundrobin (between those sub-queues that aren't overloaded acc. to
publisher confirmations); consume all simultaneously.

More tips welcome.

-- 
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/