[rabbitmq-discuss] RabbitMQ performance tips, collected with sweat and blood

[Eugene Kirpichov]

Hi,

On Fri, Aug 12, 2011 at 1:13 AM, Matthew Sackman <matthew at rabbitmq.com> wrote:
> On Thu, Aug 11, 2011 at 09:47:40PM -0700, Eugene Kirpichov wrote:
>> * Turn on prefetch. Large is good, small is better than nothing. You
>> can try a trick "reject a prefetched message if we didn't actually
>> need it for a few seconds/minutes". Consider prefetch individually for
>> each queue; e.g. it may be that you cannot turn it on for tasks, but
>> can for results.
>
> I don't know what you mean by prefetch. If you mean basic.qos then
> you're a little mistaken - basic.qos is a per channel thing, and not per
> consumer. Thus setting basic.qos affects every queue from which you're
> consuming within the same channel. Using basic.qos with non-zero values
> is known to have a fairly large negative impact on performance.
>
Yes, I meant channel indeed.
I'm surprised to hear about the negative effect; in my experiments the
positive effect was very large - basically it raised a local stress
test throughput from several thousand to several hundred thousand
messages per second; can you please clarify?

>> * Turn on autoAck
>
> autoAck and basic.qos are mutually exclusive.
That's good to know, too; thanks.

>
>> * In a cluster, always communicate with a queue only through the
>> broker that first declared it
>
> Why do you say that?
Because a queue physically exists on a single node, and accessing it
through a connection to another node causes indirect access and
traffic between the nodes; this is much slower.

>
>> * Avoid frequently doing these things, esp. on a cluster: establishing
>> a connection, BasicConsume, QueueDeclare etc. If you do them often
>> (dozens/hundreds of times per second), you're completely screwed. And
>> conversely, if you're completely screwed, the first place to look is
>> whether you're doing these things too frequently, and if you're not,
>> look again.
>
> Connection establishment certainly can be a little expensive but you
> should be able to do a few hundred a second fairly easily. Things that
> hit queues when those queues are already busy can take some time
> depending on the loading present on the queue. This is something we're
> aware of and are trying to figure out how to solve.
Yeah, I may be mistaken about connection establishment, so then the
blame is on QueueDeclare - an application of mine accidentally too
many reconnects and declares after each reconnect, and this led to all
sorts of horrible and unexplainable things.

>
>> * Increase erlang process limit
>
> We do that by default, but certainly for large installations raising
> that further may be necessary.
I started encountering this on Windows, RabbitMQ 2.5.1 when going up
from 4k consumers to 5k :)

>
>> * Split one queue into many, as a queue can only utilize 1 core -
>> across a cluster or not (either will give some benefit); publish in
>> roundrobin (between those sub-queues that aren't overloaded acc. to
>> publisher confirmations); consume all simultaneously.
>
> Yeah, the problem with that is that your message stream may require some
> sort of ordering guarantees and by publishing round-robin, you're going
> to lose that. For some use cases though, this can be an effective
> approach.
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>



-- 
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/