[rabbitmq-discuss] RabbitMQ performance tips, collected with sweat and blood

Mon Aug 15 16:33:46 BST 2011

On Mon, Aug 15, 2011 at 8:32 AM, Marek Majkowski <majek04 at gmail.com> wrote:
> On Fri, Aug 12, 2011 at 05:47, Eugene Kirpichov <ekirpichov at gmail.com> wrote:
>> I decided that I've asked too many hard or confusing questions lately,
>> and decided to contribute instead :)
>>
>> So here's what I learnt about making rabbitmq fast, or at least
>> avoiding making it slow:
>> * Turn on prefetch. Large is good, small is better than nothing. You
>> can try a trick "reject a prefetched message if we didn't actually
>> need it for a few seconds/minutes". Consider prefetch individually for
>> each queue; e.g. it may be that you cannot turn it on for tasks, but
>> can for results.
>> * Turn on autoAck
>> * Do not use transactions
>> * In a cluster, always communicate with a queue only through the
>> broker that first declared it
>> * Avoid frequently doing these things, esp. on a cluster: establishing
>> a connection, BasicConsume, QueueDeclare etc. If you do them often
>> (dozens/hundreds of times per second), you're completely screwed. And
>> conversely, if you're completely screwed, the first place to look is
>> whether you're doing these things too frequently, and if you're not,
>> look again.
>> * Avoid overloading rabbitmq with published messages in a queue and
>> making it swap like crazy. For that, use ConfirmSelect (publisher
>> confirmations), track (in your app) the number of unconfirmed messages
>> and block further submission if there are too many (more than a few
>> dozen megabytes). The effect is really dramatic.
>> * Use a 64-bit machine and raise the vm high watermark
>> * Increase erlang process limit
>> * Increase file descriptor limit
>> * Split one queue into many, as a queue can only utilize 1 core -
>> across a cluster or not (either will give some benefit); publish in
>> roundrobin (between those sub-queues that aren't overloaded acc. to
>> publisher confirmations); consume all simultaneously.
>>
>> More tips welcome.
>
> SSD disk can speed things up drastically, especially if you're using persistent
> messages and/or transactions.
Thanks, this is interesting. why so? I thought the i/o pattern of a
queueing system should be pretty sequential?

>
> Marek
>


-- 
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/