[rabbitmq-discuss] Performance tests and a warning on HIPE

[Simon MacMullen]

Thanks for a thoughtful message. On a couple of points:

On 08/03/13 03:03, McIntosh Jason wrote:
> What was surprising a bit was that this rate with both
> going was about the same rate as publishing alone.  I'd have figured
> that the disk I/O might have actually increased due to having to
> modify the database to confirm (ack) the consume.  This and the
> consumer performance tells me that consuming data from the database
> is a LOT less intensive than getting it in there.  I do want to run
> more tests to see how intensive Consumes are to the disk.

There is an optimisation in RabbitMQ that should account for this - when 
a message first reaches a queue, it is sent to the persister to go to 
disc. If the persister is busy it will join an (internal, non-AMQP) 
queue there. It can be sent out to consumers *before* it has been 
persisted - and if the consumer is autoack, or the consumer acks the 
message before it has made it to disc, we may end up never persisting 
the message at all - it's no longer relevant. Needless to say, we will 
only issue a confirm once either the message is persisted or a consumer 
has acknowledged it.

> Now for the warnings: 1)  TURN OFF HIPE.  This was testing with
> Erlang R15B03 - the latest release and RabbitMQ 3.0.2.  With HIPE and
> the load we were putting on the system, Rabbit crashes bad, no
> errors, no warning, it's just suddenly dead.  It seems to happen a
> few minutes into the load.  I've not enabled any kind of debugging to
> trace this down, but this was scary enough that we immediately turned
> off HIPE everywhere.

That certainly is scary, and unlike our experiences - we've generally 
seen HiPE crash immediately or work fine. I suspect the platform is an 
issue here. But yes, you should be cautious turning on HiPE in production...

> 2)  These numbers are very preliminary and should be taken with a
> grain of salt.  Still, it's pretty interesting to see what Rabbit can
> handle under a load, and that it seems (at least from publishing) to
> be much more IO bound than CPU bound.  I'd love to hit a smaller CPU
> system that has more IO capacity to see if I can bottle neck on CPU
> and see performance there.

Sure. Alternatively turn off persistence and you will become CPU-bound 
quite easily :-)

> 3)  Tests were done from a remote system to the server.  I used about
> 30% of a 1GB connection - there IS a possibility there's some bottle
> necking on the network due to frame sizes and other settings.
>
> 4)  I've not tested a whole lot of variations on this test yet, as
> I've not had the time, but it'd be interesting to test with different
> settings (i.e. async:1, A:0 settings, changing to XFS file system
> from EXT4, etc.).  This is currently a very default install of
> Rabbit.

At a guess, the biggest change is likely to be using an SSD.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware