[rabbitmq-discuss] RabbitMQ performance tips, collected with sweat and blood

[Marek Majkowski]

On Fri, Aug 12, 2011 at 05:47, Eugene Kirpichov <ekirpichov at gmail.com> wrote:
> I decided that I've asked too many hard or confusing questions lately,
> and decided to contribute instead :)
>
> So here's what I learnt about making rabbitmq fast, or at least
> avoiding making it slow:
> * Turn on prefetch. Large is good, small is better than nothing. You
> can try a trick "reject a prefetched message if we didn't actually
> need it for a few seconds/minutes". Consider prefetch individually for
> each queue; e.g. it may be that you cannot turn it on for tasks, but
> can for results.
> * Turn on autoAck
> * Do not use transactions
> * In a cluster, always communicate with a queue only through the
> broker that first declared it
> * Avoid frequently doing these things, esp. on a cluster: establishing
> a connection, BasicConsume, QueueDeclare etc. If you do them often
> (dozens/hundreds of times per second), you're completely screwed. And
> conversely, if you're completely screwed, the first place to look is
> whether you're doing these things too frequently, and if you're not,
> look again.
> * Avoid overloading rabbitmq with published messages in a queue and
> making it swap like crazy. For that, use ConfirmSelect (publisher
> confirmations), track (in your app) the number of unconfirmed messages
> and block further submission if there are too many (more than a few
> dozen megabytes). The effect is really dramatic.
> * Use a 64-bit machine and raise the vm high watermark
> * Increase erlang process limit
> * Increase file descriptor limit
> * Split one queue into many, as a queue can only utilize 1 core -
> across a cluster or not (either will give some benefit); publish in
> roundrobin (between those sub-queues that aren't overloaded acc. to
> publisher confirmations); consume all simultaneously.
>
> More tips welcome.

SSD disk can speed things up drastically, especially if you're using persistent
messages and/or transactions.

Marek