[rabbitmq-discuss] Response times for publisher confirms

[Aaron Pfeifer]

Hey everyone -

We've been running some tests on a RabbitMQ stack that we have running in
EC2 and encountered some unexpected behaviors in the response times for
publisher confirms and wanted to see if folks here had an idea of what the
root cause might be.

First, some details on the stack:
* EC2 m1.xlarge (15GB memory, 8 EC2 compute units, 64-bit, 1 Gigabit
Ethernet)
* Ubuntu 10.04
* Erlang R13B03
* RabbitMQ 2.8.5
* Mounted to 4 RAID0 ephemeral drives (I believe 400GB each)

...and some details on the setup:
* 16 durable, fanout exchanges
* 16 durable queues
* Each exchange mapped to a single queue
* Each message published as persistent

We have about 5,000 connections (over many servers) publishing a message
once every 15s.  Each connection has publisher confirms enabled.  Each
message is approximately 10-15KB in size (give or take).  While running
this test we tracked the following data:

* Maximum amount of time to receive a confirm: http://i.imgur.com/SrXJP.png
* Average amount of time to receive a confirm: http://i.imgur.com/ryWmr.png

Looking at this graph seems to indicate that there's something going on in
RabbitMQ about every 10 minutes that causes a significant increase in the
amount of time it takes to receive a publisher confirm.

To get some further data about what was happening during each of these
spikes, we logged the following data through vmstat and iostat on the
machine: https://gist.github.com/3874134.  You'll notice that during this
spike we experience a significant decrease in the amount of data being
written to disk and a significant increase in the await time on each disk.

And just a little background: we were planning on using durable queues and
publisher confirms to verify that the message was actually received by
RabbitMQ.  I believe I've also read that publisher confirms are typically
preferred to transactions.

Is this expected behavior for the type of test we're running?  If so, is
there anything we can do to make things a bit more consistent?

Thanks for any thoughts!

-Aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121015/e0a41547/attachment.htm>