This is the output of the same command line on the broker with high i/o:<div><br></div><div>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util</div><div>dm-0 0.00 0.00 0.00 83.62 0.00 6217.08 74.35 0.06 0.78 0.50 4.20</div>
<div><br></div><div>So you're right I was looking at blocks written per second. Is there a way to easily retrieve the average ingress rate across all queues for a broker? Maybe what's happening is a rogue client is spamming that specific server. I'd like to see the average ingress rate across all our brokers to validate that theory.</div>
<div><br></div><div>Thanks for the help troubleshooting!</div><div><br></div><div>--</div><div>Raphael.</div><div><br><div class="gmail_quote">On Wed, Oct 26, 2011 at 6:29 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com">matthias@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Raphael,<div class="im"><br>
<br>
On 27/10/11 01:47, Matthias Radestock wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Can you find out what files the 6k/s writes are going to? It can't be<br>
just the persistent message store since, as you observed, that's only<br>
writing 16M about every couple of minutes. And while I would expect to<br>
see some writes to the queue index journal files, these shouldn't add up<br>
to anywhere near 6k/s with the traffic levels you are reporting.<br>
</blockquote>
<br></div>
Btw, are you definitely looking at the correct column in the iostat output?<br>
<br>
As a test I ran 'MulticastMain -f persistent -q 1 -r 17 -s 50000' (i.e. 17msg/s, persistent, qos=1, 50k message size), and the same with "-r 8" - which roughly corresponds to the rates I saw reported on the two queues in the system with non-negligible traffic. I adjusted rabbit's memory thresholds s.t. the target_ram_count of the queues dropped to zero but w/o raising the memory alarm. 'iostat -d -x 10' shows:<br>
<br>
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util<br>
sda 0.00 219.70 0.00 57.70 0.00 2219.20 38.46 0.61 10.57 10.55 60.90<br>
<br>
So that's just 57.70 writes/s.<br><font color="#888888">
<br>
<br>
Matthias.<br>
</font></blockquote></div><br></div>