<div dir="ltr">Hi Emile, thanks for the reply.<div><br></div><div>Nothing interesting in any logs, just standard. It's difficult to pinpoint at what point these threads die (or I guess get put to sleep and never woken up again) so we can't search the logs specifically.</div>
<div><br></div><div>We're only running self-compiled RPMs because we're on RHEL5.6 and there aren't vendor-supplied RPMs. We've not made any modifications to Erlang or RabbitMQ.</div><div><br></div><div><a href="https://www.dropbox.com/s/tcvkjd87iwre093/straces.tar.gz">https://www.dropbox.com/s/tcvkjd87iwre093/straces.tar.gz</a> is a tarball of straces for one of the nodes exhibiting this issue. The largest strace is, unsurprisingly, the thread with the high load. I've set -s 1 to protect any sensitive data. The output of rabbitmqctl report is available, is there anything specific you want? It's a lot of output, and I want to be sure not to accidentally disclose sensitive data publicly. I can send you it directly though, PM me on Freenode if this would be useful.<div>
<br></div><div>etop output: <a href="https://gist.github.com/boffbowsh/116c4c2d953350111bc7">https://gist.github.com/boffbowsh/116c4c2d953350111bc7</a><br></div><div><br></div></div><div>Thanks again for the help.</div><div>
<br></div><div>Paul</div></div><div class="gmail_extra"><br clear="all"><div>Paul Bowsher</div>
<br><br><div class="gmail_quote">On Tue, Oct 8, 2013 at 1:33 PM, Emile Joubert <span dir="ltr"><<a href="mailto:emile@rabbitmq.com" target="_blank">emile@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Hi Paul,<br>
<br>
Thanks for the detailed diagnostics.<br>
<br>
On 08/10/13 11:21, Paul Bowsher wrote:<br>
<br>
> We currently have 4 nodes in this "stuck" situation on our less-critical<br>
> workloads, so we are able to provide any debugging information required.<br>
<br>
Can you supply the output of etop? e.g.<br>
etop -node rabbit@rabbit-node-name -lines 100 -sort reductions<br>
<br>
If you can get the output of "rabbitmqctl report" that will also be useful.<br>
<br>
Is there anything noteworthy in the broker or system logfiles?<br>
<br>
Can you tell what system calls are being executed by the threads with<br>
high load? e.g. "strace -pid 6017"<br>
<br>
> We're running RabbitMQ both 3.1.0 and 3.1.5 on a self-compiled RPM<br>
<br>
Were any changes made to the sources?<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
-Emile<br>
</font></span></blockquote></div><br></div>