[rabbitmq-discuss] Seeming high CPU for broker

[Simon MacMullen]

On 05/01/12 13:50, Chris Hampson wrote:
> Hi All,
>
> We've been running a few brokers for quite some time now. About a
> month ago one of them pretty much maxed out the CPU of the machine it
> was running on and has maintained this high level ever since.

Has it been restarted since?

> It's only ever using 20% (max) of the memory, not swapping, the load
> average of the machine is less than one and the service seems
> unaffected.
>
> Is this something I shouldn't be concerned with, it's not really
> having to cope with too much at the moment, a couple of active
> VHOSTS, small handful of exchanges, and in general less than 200/sec
> messages to even the busiest of those.

Well, this definitely seems wrong. One immediate possibility springs to 
mind: the mgmt plugin had some pretty poor efficiency in the face of 
large numbers of {queues/channels/connections} until 2.5.0 (both for 
responding to queries and background CPU usage). If you have lots of 
{queues/channels/connections} that's worth considering.

2.3.1 is in general a bit old, and a lot of optimisation has flowed 
under the bridge since then, so as always I'd suggest an upgrade ;-)

> I'm getting nagged by the ops guys due to it setting off
> notifications for the high CPU load, if it's expected or not
> something to worry about this I can just tell them to disable that
> notification (or at least ignore it).
>
> Are there any ways (beyond rabbitmqctl and the web interface) to see
> what's going on and if there's trouble ahead?

Nothing very lovely. Erlang ships with a utility called etop which can 
give you a top-like view of the processes in your Erlang VM. It's a bit 
fiddly and low level though.

It may not be installed by default and the script to launch it is in 
often not put in your path - for example with Debian you'd need to 
apt-get install erlang-observer and invoke it from:

/usr/lib/erlang/lib/observer-0.9.8.3/priv/bin/	

You can invoke it with:

$ /path/to/etop -node rabbit@$(hostname)

(If using our deb / rpm packaging you may need to become rabbitmq to do 
this, or copy the Erlang cookie from the rabbitmq user.)

This should show what process(es) are using all that CPU and we can go 
from there...

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware