This is the output of the same command line on the broker with high i/o:<div><br></div><div>Device: � � � � rrqm/s � wrqm/s � � r/s � � w/s � rsec/s � wsec/s avgrq-sz avgqu-sz � await �svctm �%util</div><div>dm-0 � � � � � � �0.00 � � 0.00 � �0.00 � 83.62 � � 0.00 �6217.08 � �74.35 � � 0.06 � �0.78 � 0.50 � 4.20</div>
<div><br></div><div>So you're right I was looking at blocks written per second. Is there a way to easily retrieve the average ingress rate across all queues for a broker? Maybe what's happening is a rogue client is spamming that specific server. I'd like to see the average ingress rate across all our brokers to validate that theory.</div>
<div><br></div><div>Thanks for the help troubleshooting!</div><div><br></div><div>--</div><div>Raphael.</div><div><br><div class="gmail_quote">On Wed, Oct 26, 2011 at 6:29 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com">matthias@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Raphael,<div class="im"><br>
<br>
On 27/10/11 01:47, Matthias Radestock wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Can you find out what files the 6k/s writes are going to? It can't be<br>
just the persistent message store since, as you observed, that's only<br>
writing 16M about every couple of minutes. And while I would expect to<br>
see some writes to the queue index journal files, these shouldn't add up<br>
to anywhere near 6k/s with the traffic levels you are reporting.<br>
</blockquote>
<br></div>
Btw, are you definitely looking at the correct column in the iostat output?<br>
<br>
As a test I ran 'MulticastMain -f persistent -q 1 -r 17 -s 50000' (i.e. 17msg/s, persistent, qos=1, 50k message size), and the same with "-r 8" - which roughly corresponds to the rates I saw reported on the two queues in the system with non-negligible traffic. I adjusted rabbit's memory thresholds s.t. the target_ram_count of the queues dropped to zero but w/o raising the memory alarm. 'iostat -d -x 10' shows:<br>
<br>
Device: � � � � rrqm/s � wrqm/s � � r/s � � w/s � rsec/s � wsec/s avgrq-sz avgqu-sz � await �svctm �%util<br>
sda � � � � � � � 0.00 � 219.70 � �0.00 � 57.70 � � 0.00 �2219.20 38.46 � � 0.61 � 10.57 �10.55 �60.90<br>
<br>
So that's just 57.70 writes/s.<br><font color="#888888">
<br>
<br>
Matthias.<br>
</font></blockquote></div><br></div>