<div dir="ltr">Hi Matthias,<div><br></div><div>Thanks for the response, so the extra process queue during our peak message throughput which in turn causes the load average to be reported much higher is just a side effect of this? We normally have about 600 connections (mostly consumers) and our producers all publish to topic exchanges, we have around 250 queues.</div>
<div><br></div><div>Regards</div><div>Dan</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 19, 2013 at 6:52 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dan,<div class="im"><br>
<br>
On 19/08/13 03:58, Dan_b wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
We have recently upgraded from rabbitmq-server 2.6.1 to 3.1.x (we have one<br>
cluster running 3.1.5 and one running 3.1.3). On both servers we have seen<br>
the load average increase significantly, with it peaking up to 2x the number<br>
cores. The CPU utilization and IO wait aren't affected and it seems to be<br>
down to internal locks.<br>
</blockquote>
<br></div>
I'd guess those FUTEXes have something to do with timers. They are immediately followed by fsyncs, which in rabbit are typically triggered by timers. So the sequence you are seeing is<br>
<br>
1. write message to disk<br>
2. wait for timer to expire<br>
3. fsync<br>
<br>
e.g.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
write(27,<br>
"\0\0\0\0\0\3\324=\226(\16\<u></u>265\355\200\307\7J\34b\4\240\"<u></u>\355)\0\0\0\0\0\0\0\0"...,<br>
48) = 48<br>
futex(0x2aaaab594524, FUTEX_WAIT_PRIVATE, 235521, NULL) = -1 EAGAIN<br>
(Resource temporarily unavailable)<br>
futex(0x2aaaab5944f8, FUTEX_WAKE_PRIVATE, 1) = 0<br>
fsync(27) = 0<br>
</blockquote>
<br></div>
So nothing to worry about.<span class="HOEnZb"><font color="#888888"><br>
<br>
Matthias.<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div style="font-family:Verdana,Helvetica,Arial,sans-serif;font-size:13px;line-height:16px;text-align:left;background-color:rgb(255,255,255)">
<b>Dan Bason </b></div><div style="font-family:Verdana,Helvetica,Arial,sans-serif;font-size:13px;line-height:16px;text-align:left;background-color:rgb(255,255,255)">Infrastructure Engineer – Telogis </div><div style="font-family:Verdana,Helvetica,Arial,sans-serif;font-size:13px;line-height:16px;text-align:left;background-color:rgb(255,255,255)">
<div style="font-family:'Open Sans',sans-serif"><a href="http://www.telogis.com" target="_blank">www.telogis.com</a> <a href="http://www.telogis.co.nz" target="_blank">www.telogis.co.nz</a> </div><div style="font-family:'Open Sans',sans-serif">
+1 201 477-1448 (USA) +64 3 281-8065 (NZ) </div><div style="font-family:'Open Sans',sans-serif"><br></div><div style="font-family:'Open Sans',sans-serif"><p style="font-family:arial,sans-serif;font-size:12.727272033691406px;line-height:normal">
</p><p style="margin:0px"><a href="http://www.google.com/url?q=http%3A%2F%2Fwww.telogis.com%2F&sa=D&sntz=1&usg=AFrqEze6z4qIWKcAMjgXqM25xkuw9JMysQ" style="font-family:arial;font-size:small;color:rgb(74,146,193)!important" target="_blank"><img src="http://www.telogis.com/images/telogis-logo-black-sm.png" style="border:none;padding:0px"></a></p>
<br style="color:rgb(0,0,0);font-family:arial;font-size:small"><i style="color:rgb(0,0,0);font-family:arial;font-size:small">One Platform for Location Intelligence</i><span style="color:rgb(0,0,0);font-family:arial;font-size:small"> </span><br>
<p></p></div></div></div>
</div>
<br>
<p><span style="color:gray"><font size="1">This e-mail,
and any attachments, is intended only for use by the addressee(s) named herein
and may contain legally privileged and/or confidential information. It is
the property of Telogis. If you are not the intended recipient of this
e-mail, you are hereby notified that any dissemination, distribution or copying
of this e-mail, any attachments thereto, and use of the information contained,
is strictly prohibited. If you have received this e-mail in error, please
notify the sender and permanently delete the original and any copy thereof.</font></span><span style="font-size:1.3em;font-size:1.3em"></span></p>