[rabbitmq-discuss] Performance tests and a warning on HIPE
simon at rabbitmq.com
Fri Mar 8 13:40:33 GMT 2013
Thanks for a thoughtful message. On a couple of points:
On 08/03/13 03:03, McIntosh Jason wrote:
> What was surprising a bit was that this rate with both
> going was about the same rate as publishing alone. I'd have figured
> that the disk I/O might have actually increased due to having to
> modify the database to confirm (ack) the consume. This and the
> consumer performance tells me that consuming data from the database
> is a LOT less intensive than getting it in there. I do want to run
> more tests to see how intensive Consumes are to the disk.
There is an optimisation in RabbitMQ that should account for this - when
a message first reaches a queue, it is sent to the persister to go to
disc. If the persister is busy it will join an (internal, non-AMQP)
queue there. It can be sent out to consumers *before* it has been
persisted - and if the consumer is autoack, or the consumer acks the
message before it has made it to disc, we may end up never persisting
the message at all - it's no longer relevant. Needless to say, we will
only issue a confirm once either the message is persisted or a consumer
has acknowledged it.
> Now for the warnings: 1) TURN OFF HIPE. This was testing with
> Erlang R15B03 - the latest release and RabbitMQ 3.0.2. With HIPE and
> the load we were putting on the system, Rabbit crashes bad, no
> errors, no warning, it's just suddenly dead. It seems to happen a
> few minutes into the load. I've not enabled any kind of debugging to
> trace this down, but this was scary enough that we immediately turned
> off HIPE everywhere.
That certainly is scary, and unlike our experiences - we've generally
seen HiPE crash immediately or work fine. I suspect the platform is an
issue here. But yes, you should be cautious turning on HiPE in production...
> 2) These numbers are very preliminary and should be taken with a
> grain of salt. Still, it's pretty interesting to see what Rabbit can
> handle under a load, and that it seems (at least from publishing) to
> be much more IO bound than CPU bound. I'd love to hit a smaller CPU
> system that has more IO capacity to see if I can bottle neck on CPU
> and see performance there.
Sure. Alternatively turn off persistence and you will become CPU-bound
quite easily :-)
> 3) Tests were done from a remote system to the server. I used about
> 30% of a 1GB connection - there IS a possibility there's some bottle
> necking on the network due to frame sizes and other settings.
> 4) I've not tested a whole lot of variations on this test yet, as
> I've not had the time, but it'd be interesting to test with different
> settings (i.e. async:1, A:0 settings, changing to XFS file system
> from EXT4, etc.). This is currently a very default install of
At a guess, the biggest change is likely to be using an SSD.
More information about the rabbitmq-discuss