<div dir="ltr">On Thu, Jul 11, 2013 at 2:08 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>></span> wrote:<div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">
On 11/07/13 17:38, Kevin Goess wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Circling back after an earlier question about this problem. We've<br>
upgraded to 3.1.3-1. We've moved the mnesia files to a dedicated 10k<br>
sata drive via RABBITMQ_MNESIA_BASE, a new drive that nothing else is<br>
using, on a xen virtual machine that is only running rabbitmq. We<br>
generally deal with about 20 messages per second.<br>
<br>
But we're seeing pretty heavy disk activity, with iowait 30-40%, and<br>
iostat numbers like this:<br>
<br>
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s<br>
avgrq-sz avgqu-sz await svctm %util<br>
xvdbp1 0.00 60.00 0.00 143.40 0.00 1452.80<br>
10.13 0.88 <a href="tel:6.14%20%20%204.22%20%2060.48" value="+16144226048" target="_blank">6.14 4.22 60.48</a><br>
<br>
Is this expected? That seems excessive, and I worry that the high<br>
iowait could cause us problems as we grow, so I was hoping to get some<br>
pointers to things to look at that to improve that.<br>
</blockquote>
<br></div>
My suspicious is that somehow your usage pattern results in a lot of queue/exchange/binding creation/deletion, which, if the entities involved are durable, require disk writes and fsyncs.<br>
<br>
To test the validity of that theory, please try to find out what files are seeing activity, and report back.<br></blockquote><div><br></div><div><br></div><div>This is all the writing that's gone on in the last ten minutes:</div>
<div><br></div><div><div>$ find . -newer kg-test-timer -type f -ls</div><div>262153 10784 -rw-r--r-- 1 root root 11036923 Jul 16 12:31 ./msg_store_persistent/41228.rdq</div><div>262155 16392 -rw-r--r-- 1 root root 16777655 Jul 16 12:21 ./msg_store_persistent/41227.rdq</div>
<div>262274 120 -rw-r--r-- 1 rabbitmq rabbitmq 116640 Jul 16 12:30 ./queues/8EP2UVB3Z8SAIWFBFKQ0100K4/journal.jif</div><div>262166 32 -rw-r--r-- 1 root root 28222 Jul 16 12:28 ./queues/9K0C0VGF04K4O788HH6ZFMDAG/4747.idx</div>
<div>262204 24 -rw-r--r-- 1 rabbitmq rabbitmq 20000 Jul 16 12:30 ./queues/9K0C0VGF04K4O788HH6ZFMDAG/journal.jif</div><div>262228 0 -rw-r--r-- 1 rabbitmq rabbitmq 0 Jul 16 12:27 ./queues/A6K4C7JLF1P0AHEXXN9633747/journal.jif</div>
<div>262172 4 -rw-r--r-- 1 root root 28 Jul 16 12:26 ./queues/C8NMZYECUT2TCZUNODNUQ122Y/0.idx</div><div>262224 0 -rw-r--r-- 1 rabbitmq rabbitmq 0 Jul 16 12:26 ./queues/C8NMZYECUT2TCZUNODNUQ122Y/journal.jif</div>
<div>262291 136 -rw-r--r-- 1 rabbitmq rabbitmq 134008 Jul 16 12:31 ./queues/5W6SPLYHNTA4ZH2CONMRXOT90/journal.jif</div><div>262161 12 -rw-r--r-- 1 root root 8256 Jul 16 12:30 ./queues/3GMBIL72D6Q6F4LC6TWTYCPWA/1.idx</div>
<div>262236 0 -rw-r--r-- 1 rabbitmq rabbitmq 0 Jul 16 12:30 ./queues/3GMBIL72D6Q6F4LC6TWTYCPWA/journal.jif</div><div>262234 424 -rw-r--r-- 1 rabbitmq rabbitmq 430024 Jul 16 12:31 ./queues/ERZ5DFKEUXBK1QJQJLBQN3KFI/journal.jif</div>
<div>262199 528 -rw-r--r-- 1 rabbitmq rabbitmq 536208 Jul 16 12:30 ./queues/DWEIOI0J2S6CMT74JYWUZXVL8/journal.jif</div><div>262282 28 -rw-r--r-- 1 rabbitmq rabbitmq 21600 Jul 16 12:30 ./queues/1XAUC81LT5GM7ZI05RROF4KL0/journal.jif</div>
<div>262297 0 -rw-r--r-- 1 rabbitmq rabbitmq 0 Jul 16 12:19 ./queues/31J4BX7549NFNZ1Q6XH1Z19MJ/journal.jif</div></div><div><br></div><div><br></div><div>and that list is almost identical to that for the last *hour*, with the addition of these files</div>
<div><br></div><div><div>262152 80 -rw-r--r-- 1 rabbitmq rabbitmq 78942 Jul 16 11:53 ./msg_store_persistent/17778.rdq</div></div><div><div>262321 0 -rw-r--r-- 1 rabbitmq rabbitmq 0 Jul 16 11:36 ./queues/AVS8NDWHNCZ8X0Y2599HSOVJ0/journal.jif</div>
<div>262341 0 -rw-r--r-- 1 rabbitmq rabbitmq 0 Jul 16 11:42 ./queues/U7FN5251IRUUH23XQIPFTBND/journal.jif</div></div><div><br></div><div><br></div><div>Does that mean anything to you? Is there a way we can match those directory names with queues?</div>
<div><br></div><div><br></div></div>
</div></div>