Hey everyone -<div><br></div><div>We've been running some tests on a RabbitMQ stack that we have running in EC2 and encountered some unexpected behaviors in the response times for publisher confirms and wanted to see if folks here had an idea of what the root cause might be.</div>
<div><br></div><div>First, some details on the stack:</div><div>* EC2 m1.xlarge (15GB memory, 8 EC2 compute units, 64-bit, 1 Gigabit Ethernet)</div><div>* Ubuntu 10.04</div><div>* Erlang R13B03</div><div><div>* RabbitMQ 2.8.5</div>
<div>* Mounted to 4 RAID0 ephemeral drives (I believe 400GB each)</div>
</div><div><br></div><div>...and some details on the setup:</div><div>* 16 durable, fanout exchanges</div><div>* 16 durable queues</div><div>* Each exchange mapped to a single queue</div><div>* Each message published as persistent</div>
<div><br></div><div>We have about 5,000 connections (over many servers) publishing a message once every 15s. Each connection has publisher confirms enabled. Each message is approximately 10-15KB in size (give or take). While running this test we tracked the following data:</div>
<div>
<br></div><div>* Maximum amount of time to receive a confirm: <a href="http://i.imgur.com/SrXJP.png" target="_blank">http://i.imgur.com/SrXJP.png</a></div><div>* Average amount of time to receive a confirm: <a href="http://i.imgur.com/ryWmr.png" target="_blank">http://i.imgur.com/ryWmr.png</a></div>
<div><br></div><div>Looking at this graph seems to indicate that there's something going on in RabbitMQ about every 10 minutes that causes a significant increase in the amount of time it takes to receive a publisher confirm.</div>
<div><br></div><div>To get some further data about what was happening during each of these spikes, we logged the following data through vmstat and iostat on the machine: <a href="https://gist.github.com/3874134">https://gist.github.com/3874134</a>. You'll notice that during this spike we experience a significant decrease in the amount of data being written to disk and a significant increase in the await time on each disk.</div>
<div><br></div><div>And just a little background: we were planning on using durable queues and publisher confirms to verify that the message was actually received by RabbitMQ. I believe I've also read that publisher confirms are typically preferred to transactions.</div>
<div><br></div><div>Is this expected behavior for the type of test we're running? If so, is there anything we can do to make things a bit more consistent?</div><div><br></div><div>Thanks for any thoughts!</div><div>
<br>
</div><div>-Aaron</div>