[rabbitmq-discuss] rabbitmq-server process consumes 100% of 1 CPU (rabbitmq 2.8.4 on Erlang R14B04)

[Ceri Storey]

Hi Emile,

I'm one of Anthony's Colleagues here at LShift, who assisted in the
initial investigation.

    Thanks for this very detailed diagnostic information. Could you perhaps
    supply the output of "rabbitmqctl report" as well? Feel free to compress
    and send to me directly if it is too large.

Size of the report isn't the main issue in this case--sadly, it looks
like even the process of getting a queue listing wasn't going to
terminate any time soon (I gave up after ~10 minutes). I've attached the
output from a broken rabbit (although that only lists "rabbitmqctl
status") and a report from the same rabbit with an increased file
descriptor limit.

     I have not been able to reproduce the problem - does this happen each
    time you start the cluster? does stopping and starting the slave nodes
    make any difference? Have you discovered a sequence of actions that is
    likely to reproduce the problem?

In the example I'm seeing (on our internal build slaves that the earlier
etop results were derived from), we have about 1400 queues left over
from a set of rather unhygenic integration tests, so I'd infer that
there's some kind of pathological interaction between the mnesia queue
configuration database resource management and the file_handle_cache
module. So, it does indeed happen every time we restart the single
instance (on our CI machines, they're single instances).

On this hunch, I've bumped our file descriptor limit on one machine to
8192 from 1024 files, and it does indeed that the resource usage does
drop dramatically once the instance has started up. So, I think that
brings us to a work around for one case, if not an explanation of the
underlying behaviour.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120712/739f6a02/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbitmqctl-report-spinning.txt.xz
Type: application/x-xz
Size: 1244 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120712/739f6a02/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbitmqctl-report-stable.txt.xz
Type: application/x-xz
Size: 27020 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120712/739f6a02/attachment-0001.bin>