Running that script produces this:<div><br><div><div>[{490415,{rabbit@queue01,os_pid}},</div><div>�{7465,{rabbit@queue02,os_pid}},</div><div>�{470,{rabbit@queue01,mem_ets}},</div><div>�{912,{rabbit@queue02,mem_ets}},</div>
<div>�{249,{rabbit@queue01,mem_binary}},</div><div>�{584,{rabbit@queue02,mem_binary}},</div><div>�{173,{rabbit@queue01,mem_proc}},</div><div>�{660,{rabbit@queue02,mem_proc}},</div><div>�{172,{rabbit@queue01,mem_proc_used}},</div>
<div>�{555,{rabbit@queue02,mem_proc_used}},</div><div>�{144,{rabbit@queue01,mem_atom}},</div><div>�{1066,{rabbit@queue02,mem_atom}},</div><div>�{256,{rabbit@queue01,mem_atom_used}},</div><div>�{716,{rabbit@queue02,mem_atom_used}},</div>
<div>�{178,{rabbit@queue01,mem_code}},</div><div>�{474,{rabbit@queue02,mem_code}},</div><div>�{62,{rabbit@queue01,fd_used}},</div><div>�{422,{rabbit@queue02,fd_used}},</div><div>�{62,{rabbit@queue01,fd_total}},</div><div>
�{437,{rabbit@queue02,fd_total}},</div><div>�{76,{rabbit@queue01,sockets_used}},</div><div>�{446,{rabbit@queue02,sockets_used}},</div><div>�{78,{rabbit@queue01,sockets_total}},</div><div>�{678,{rabbit@queue02,sockets_total}},</div>
<div>�{181,{rabbit@queue01,mem_used}},</div><div>�{1126,{rabbit@queue02,mem_used}},</div><div>�{71,{rabbit@queue01,mem_limit}},</div><div>�{7747,{rabbit@queue02,mem_limit}},</div><div>�{94,{rabbit@queue01,mem_alarm}},</div>
<div>�{636,{rabbit@queue02,mem_alarm}},</div><div>�{87,{rabbit@queue01,disk_free_limit}},</div><div>�{588,{rabbit@queue02,disk_free_limit}},</div><div>�{1152028,{rabbit@queue01,disk_free}},</div><div>�{1694075,{rabbit@queue02,disk_free}},</div>
<div>�{635870,{rabbit@queue01,disk_free_alarm}},</div><div>�{310155,{rabbit@queue02,disk_free_alarm}},</div><div>�{999554,{rabbit@queue01,proc_used}},</div><div>�{823,{rabbit@queue02,proc_used}},</div><div>�{1286885,{rabbit@queue01,proc_total}},</div>
<div>�{27490,{rabbit@queue02,proc_total}},</div><div>�{1207,{rabbit@queue01,statistics_level}},</div><div>�{960135,{rabbit@queue02,statistics_level}},</div><div>�{70,{rabbit@queue01,erlang_version}},</div><div>�{956790,{rabbit@queue02,erlang_version}},</div>
<div>�{47244,{rabbit@queue01,uptime}},</div><div>�{2882,{rabbit@queue02,uptime}},</div><div>�{949552,{rabbit@queue01,run_queue}},</div><div>�{93713,{rabbit@queue02,run_queue}},</div><div>�{974491,{rabbit@queue01,processors}},</div>
<div>�{789,{rabbit@queue02,processors}},</div><div>�{99,{rabbit@queue01,exchange_types}},</div><div>�{938,{rabbit@queue02,exchange_types}},</div><div>�{87,{rabbit@queue01,auth_mechanisms}},</div><div>�{428,{rabbit@queue02,auth_mechanisms}},</div>
<div>�{153,{rabbit@queue01,applications}},</div><div>�{585,{rabbit@queue02,applications}}]</div><div>...done.</div><div><br></div><div>Seems like the big values are:</div><div>{490415,{rabbit@queue01,os_pid}}</div><div>{1152028,{rabbit@queue01,disk_free}}</div>
<div>{1694075,{rabbit@queue02,disk_free}}</div><div>{635870,{rabbit@queue01,disk_free_alarm}}</div><div>{310155,{rabbit@queue02,disk_free_alarm}}</div><div>{1286885,{rabbit@queue01,proc_total}}</div>{960135,{rabbit@queue02,statistics_level}}</div>
<div>{956790,{rabbit@queue02,erlang_version}}</div><div>{949552,{rabbit@queue01,run_queue}}</div><div>{93713,{rabbit@queue02,run_queue}}</div><div>{974491,{rabbit@queue01,processors}}</div><div><br></div><div>The worst offenders are DiskFree on both servers and�proc_total on queue01.</div>
<div><br></div><div>I'm not sure what I should be l looking for here.</div><div><br></div><div>Again! Thanks so much for your help!</div><div><br></div><div>-Dave</div><div><br><div class="gmail_quote">On Fri, Oct 19, 2012 at 12:58 PM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 19/10/12 17:51, Dave Seltzer wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In terms of looking for the cause of the slowness is this usually<br>
related to DiskIO, or am I looking at a CPU issue.<br>
</blockquote>
<br>
Depends on what's slow!<br>
<br>
Let's iterate again:<br>
<br>
rabbitmqctl eval '[timer:tc(fun () -> rabbit_mgmt_external_stats:<u></u>info(N, [K]), {N, K} end) || K <- [os_pid,mem_ets,mem_binary,<u></u>mem_proc,mem_proc_used,mem_<u></u>atom,mem_atom_used,mem_code,<u></u>fd_used,fd_total,sockets_used,<u></u>sockets_total,mem_used,mem_<u></u>limit,mem_alarm,disk_free_<u></u>limit,disk_free,disk_free_<u></u>alarm,proc_used,proc_total,<u></u>statistics_level,erlang_<u></u>version,uptime,run_queue,<u></u>processors,exchange_types,<u></u>auth_mechanisms,applications], N <- [rabbit@queue01,rabbit@queue02<u></u>]].'<br>
<br>
This will provide a time for each individual info item.<br>
<br>
Cheers, Simon<span class="HOEnZb"><font color="#888888"><br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><span style="font-size:x-small"><a href="mailto:dseltzer@tveyes.com" target="_blank">Dave Seltzer</a></span></div><div><font size="1">Chief Systems Architect</font></div>
<div><font size="1">TVEyes</font></div><div><font size="1">(203) 254-3600 x222</font></div><br>
</div></div>