[rabbitmq-discuss] "Dead" beam.smp threads under high load

[Paul Bowsher]

Hi Emile, thanks for the reply.

Nothing interesting in any logs, just standard. It's difficult to pinpoint
at what point these threads die (or I guess get put to sleep and never
woken up again) so we can't search the logs specifically.

We're only running self-compiled RPMs because we're on RHEL5.6 and there
aren't vendor-supplied RPMs. We've not made any modifications to Erlang or
RabbitMQ.

https://www.dropbox.com/s/tcvkjd87iwre093/straces.tar.gz is a tarball of
straces for one of the nodes exhibiting this issue. The largest strace is,
unsurprisingly, the thread with the high load. I've set -s 1 to protect any
sensitive data. The output of rabbitmqctl report is available, is there
anything specific you want? It's a lot of output, and I want to be sure not
to accidentally disclose sensitive data publicly. I can send you it
directly though, PM me on Freenode if this would be useful.

etop output: https://gist.github.com/boffbowsh/116c4c2d953350111bc7

Thanks again for the help.

Paul

Paul Bowsher


On Tue, Oct 8, 2013 at 1:33 PM, Emile Joubert <emile at rabbitmq.com> wrote:

>
> Hi Paul,
>
> Thanks for the detailed diagnostics.
>
> On 08/10/13 11:21, Paul Bowsher wrote:
>
> > We currently have 4 nodes in this "stuck" situation on our less-critical
> > workloads, so we are able to provide any debugging information required.
>
> Can you supply the output of etop? e.g.
>  etop -node rabbit at rabbit-node-name -lines 100 -sort reductions
>
> If you can get the output of "rabbitmqctl report" that will also be useful.
>
> Is there anything noteworthy in the broker or system logfiles?
>
> Can you tell what system calls are being executed by the threads with
> high load? e.g. "strace -pid 6017"
>
> > We're running RabbitMQ both 3.1.0 and 3.1.5 on a self-compiled RPM
>
> Were any changes made to the sources?
>
>
>
> -Emile
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131008/4cf40acd/attachment.htm>