<div dir="ltr">On Thu, Jul 11, 2013 at 2:08 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>></span> wrote:<div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">
On 11/07/13 17:38, Kevin Goess wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Circling back after an earlier question about this problem. We've<br>
upgraded to 3.1.3-1. �We've moved the mnesia files to a dedicated 10k<br>
sata drive via RABBITMQ_MNESIA_BASE, a new drive that nothing else is<br>
using, on a xen virtual machine that is only running rabbitmq. �We<br>
generally deal with about 20 messages per second.<br>
<br>
But we're seeing pretty heavy disk activity, with iowait 30-40%, and<br>
iostat numbers like this:<br>
<br>
Device: � � � � rrqm/s � wrqm/s � � r/s � � w/s � rsec/s � wsec/s<br>
avgrq-sz avgqu-sz � await �svctm �%util<br>
xvdbp1 � � � � � �0.00 � �60.00 � �0.00 �143.40 � � 0.00 �1452.80<br>
� 10.13 � � 0.88 � �<a href="tel:6.14%20%20%204.22%20%2060.48" value="+16144226048" target="_blank">6.14 4.22 60.48</a><br>
<br>
Is this expected? �That seems excessive, and I worry that the high<br>
iowait could cause us problems as we grow, so I was hoping to get some<br>
pointers to things to look at that to improve that.<br>
</blockquote>
<br></div>
My suspicious is that somehow your usage pattern results in a lot of queue/exchange/binding creation/deletion, which, if the entities involved are durable, require disk writes and fsyncs.<br>
<br>
To test the validity of that theory, please try to find out what files are seeing activity, and report back.<br></blockquote><div><br></div><div><br></div><div>This is all the writing that's gone on in the last ten minutes:</div>
<div><br></div><div><div>$ find . -newer kg-test-timer -type f -ls</div><div>262153 10784 -rw-r--r-- � 1 root � � root � � 11036923 Jul 16 12:31 ./msg_store_persistent/41228.rdq</div><div>262155 16392 -rw-r--r-- � 1 root � � root � � 16777655 Jul 16 12:21 ./msg_store_persistent/41227.rdq</div>
<div>262274 �120 -rw-r--r-- � 1 rabbitmq rabbitmq � 116640 Jul 16 12:30 ./queues/8EP2UVB3Z8SAIWFBFKQ0100K4/journal.jif</div><div>262166 � 32 -rw-r--r-- � 1 root � � root � � � �28222 Jul 16 12:28 ./queues/9K0C0VGF04K4O788HH6ZFMDAG/4747.idx</div>
<div>262204 � 24 -rw-r--r-- � 1 rabbitmq rabbitmq � �20000 Jul 16 12:30 ./queues/9K0C0VGF04K4O788HH6ZFMDAG/journal.jif</div><div>262228 � �0 -rw-r--r-- � 1 rabbitmq rabbitmq � � � �0 Jul 16 12:27 ./queues/A6K4C7JLF1P0AHEXXN9633747/journal.jif</div>
<div>262172 � �4 -rw-r--r-- � 1 root � � root � � � � � 28 Jul 16 12:26 ./queues/C8NMZYECUT2TCZUNODNUQ122Y/0.idx</div><div>262224 � �0 -rw-r--r-- � 1 rabbitmq rabbitmq � � � �0 Jul 16 12:26 ./queues/C8NMZYECUT2TCZUNODNUQ122Y/journal.jif</div>
<div>262291 �136 -rw-r--r-- � 1 rabbitmq rabbitmq � 134008 Jul 16 12:31 ./queues/5W6SPLYHNTA4ZH2CONMRXOT90/journal.jif</div><div>262161 � 12 -rw-r--r-- � 1 root � � root � � � � 8256 Jul 16 12:30 ./queues/3GMBIL72D6Q6F4LC6TWTYCPWA/1.idx</div>
<div>262236 � �0 -rw-r--r-- � 1 rabbitmq rabbitmq � � � �0 Jul 16 12:30 ./queues/3GMBIL72D6Q6F4LC6TWTYCPWA/journal.jif</div><div>262234 �424 -rw-r--r-- � 1 rabbitmq rabbitmq � 430024 Jul 16 12:31 ./queues/ERZ5DFKEUXBK1QJQJLBQN3KFI/journal.jif</div>
<div>262199 �528 -rw-r--r-- � 1 rabbitmq rabbitmq � 536208 Jul 16 12:30 ./queues/DWEIOI0J2S6CMT74JYWUZXVL8/journal.jif</div><div>262282 � 28 -rw-r--r-- � 1 rabbitmq rabbitmq � �21600 Jul 16 12:30 ./queues/1XAUC81LT5GM7ZI05RROF4KL0/journal.jif</div>
<div>262297 � �0 -rw-r--r-- � 1 rabbitmq rabbitmq � � � �0 Jul 16 12:19 ./queues/31J4BX7549NFNZ1Q6XH1Z19MJ/journal.jif</div></div><div><br></div><div><br></div><div>and that list is almost identical to that for the last *hour*, with the addition of these files</div>
<div><br></div><div><div>262152 � 80 -rw-r--r-- � 1 rabbitmq rabbitmq � �78942 Jul 16 11:53 ./msg_store_persistent/17778.rdq</div></div><div><div>262321 � �0 -rw-r--r-- � 1 rabbitmq rabbitmq � � � �0 Jul 16 11:36 ./queues/AVS8NDWHNCZ8X0Y2599HSOVJ0/journal.jif</div>
<div>262341 � �0 -rw-r--r-- � 1 rabbitmq rabbitmq � � � �0 Jul 16 11:42 ./queues/U7FN5251IRUUH23XQIPFTBND/journal.jif</div></div><div><br></div><div><br></div><div>Does that mean anything to you? �Is there a way we can match those directory names with queues?</div>
<div><br></div><div><br></div></div>
</div></div>