<html><body bgcolor="#FFFFFF"><div>DTrace is pretty amazing. We've been using it for troubleshooting MySQL since 2006 (OpenSolaris/OpenIndiana). Once you've used it, it's hard to live without. Strace doesn't even hold a candle.&nbsp;</div><div><br></div><div>-J<br><br>Sent via iPhone<div><br></div><div>Is your email Premiere?</div></div><div><br>On Sep 15, 2011, at 10:11, Alvaro Videla &lt;<a href="mailto:videlalvaro@gmail.com">videlalvaro@gmail.com</a>&gt; wrote:<br><br></div><div></div><blockquote type="cite"><div>I hope this DTrace branch gets merged with the default Erlang. I think is quite useful<br><br><div class="gmail_quote">On Thu, Sep 15, 2011 at 6:00 PM, Jason J. W. Williams <span dir="ltr">&lt;<a href="mailto:jasonjwwilliams@gmail.com"><a href="mailto:jasonjwwilliams@gmail.com">jasonjwwilliams@gmail.com</a></a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#FFFFFF"><div>OpenIndiana and &nbsp;(open Solaris) are great Rabbit platforms. FreeBSD has DTrace too. DTrace for Linux isn't up to snuff with the other platforms yet, but it's still worlds ahead of SystemTap.&nbsp;</div>
<div><br></div><div>-J<br><br>Sent via iPhone<div><br></div><div>Is your email Premiere?</div></div><div><div></div><div class="h5"><div><br>On Sep 15, 2011, at 8:03, Alvaro Videla &lt;<a href="mailto:videlalvaro@gmail.com" target="_blank"><a href="mailto:videlalvaro@gmail.com">videlalvaro@gmail.com</a></a>&gt; wrote:<br>
<br></div><div></div><blockquote type="cite"><div><a href="https://twitter.com/hylomorphism/status/114000680875851776" target="_blank"></a><a href="https://twitter.com/hylomorphism/status/114000680875851776" target="_blank"></a><a href="https://twitter.com/hylomorphism/status/114000680875851776" target="_blank"></a><a href="https://twitter.com/hylomorphism/status/114000680875851776" target="_blank"></a><a href="https://twitter.com/hylomorphism/status/114000680875851776" target="_blank"></a><div>

Hi,</div><div><br></div><div>I saw Matthew tweeting "Oh RabbitMQ, why are you taking over an hour to recover 100k queues? What /are/ you doing?!"&nbsp;<a href="https://twitter.com/hylomorphism/status/114000680875851776" target="_blank"></a><a href="https://twitter.com/hylomorphism/status/114000680875851776" target="_blank"><a href="https://twitter.com/hylomorphism/status/114000680875851776">https://twitter.com/hylomorphism/status/114000680875851776</a></a>&nbsp;</div>

<div><br></div><div>I've been doing some stuff at work with DTrace and I think is pretty cool when it comes to find out what a process is actually doing.</div><div><br></div><div>At the same time there's an experimental OTP fork on Github that adds Dtrace probes to Erlang:&nbsp;<a href="https://github.com/dustin/otp/wiki/DTrace" target="_blank"></a><a href="https://github.com/dustin/otp/wiki/DTrace" target="_blank"><a href="https://github.com/dustin/otp/wiki/DTrace">https://github.com/dustin/otp/wiki/DTrace</a></a></div>

<div><br></div><div>See here for the available probes:&nbsp;<a href="https://github.com/dustin/otp/blob/dtrace/erts/emulator/beam/erlang_dtrace.d" target="_blank"></a><a href="https://github.com/dustin/otp/blob/dtrace/erts/emulator/beam/erlang_dtrace.d" target="_blank"><a href="https://github.com/dustin/otp/blob/dtrace/erts/emulator/beam/erlang_dtrace.d">https://github.com/dustin/otp/blob/dtrace/erts/emulator/beam/erlang_dtrace.d</a></a></div>

<div><br></div><div>So for example running RabbitMQ based on that Erlang branch you could do things like in this gist:&nbsp;<a href="https://gist.github.com/1219266" target="_blank"></a><a href="https://gist.github.com/1219266" target="_blank"><a href="https://gist.github.com/1219266">https://gist.github.com/1219266</a></a></div>
<div><br></div><div>
<span style="font-family:helvetica, arial, freesans, clean, sans-serif;font-size:11px;line-height:14px"><pre style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em;font-family:'Bitstream Vera Sans Mono', Courier, monospace"><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em"><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">erlang</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">$</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">target</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">:::</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em;font-weight:bold">function</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">-</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">entry</span></div>

<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em"><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">/</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">copyinstr</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">(</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">arg1</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">)</span> <span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">==</span> <span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em;color:rgb(221, 17, 68)">"rabbit_exchange_type_direct:route/2"</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">/</span></div>

<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em"><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">{</span></div>

<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">&nbsp;&nbsp;&nbsp;&nbsp;<span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">printf</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">(</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em;color:rgb(221, 17, 68)">"%s %s"</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">,</span> <span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">probemod</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">,</span> <span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">probefunc</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">);</span></div>

<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">&nbsp;&nbsp;&nbsp;&nbsp;<span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">self</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">-&gt;</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">traceIt</span> <span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">=</span> <span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em;color:rgb(0, 153, 153)">1</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">;</span></div>

<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em"><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">}</span></div>

</pre></span></div><div><br></div><div>There I enable the flag traceIt only when route/2 is called on the direct exchange.</div><div><br></div><div>From there is just a matter of adding probes to different syscalls or io access etc based using patterns on that flag like:</div>

<div><span style="font-family:'Bitstream Vera Sans Mono', Courier, monospace;font-size:12px;line-height:16px;white-space:pre-wrap"><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em"><br>

</span></span></div><div><span style="font-family:'Bitstream Vera Sans Mono', Courier, monospace;font-size:12px;line-height:16px;white-space:pre-wrap"><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">erlang</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">$</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">target</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">:::</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em;font-weight:bold">function</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">-</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">entry</span></span></div>

<div><span style="font-family:'Bitstream Vera Sans Mono', Courier, monospace;font-size:12px;line-height:16px;white-space:pre-wrap"><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">/</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">self</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">-&gt;</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">traceIt</span><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">/</span></span></div>

<div><span style="font-family:'Bitstream Vera Sans Mono', Courier, monospace;font-size:12px;line-height:16px;white-space:pre-wrap"><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">{</span></span></div>

<div><span style="font-family:'Bitstream Vera Sans Mono', Courier, monospace;font-size:12px;line-height:16px;white-space:pre-wrap"><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em"> //do something here</span></span></div>

<div><span style="font-family:'Bitstream Vera Sans Mono', Courier, monospace;font-size:12px;line-height:16px;white-space:pre-wrap"><span style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em">}</span></span></div>

<div><br></div><div>and so on.&nbsp;</div><div><br></div><div>You can run the gist like:&nbsp;<span style="color:rgb(51, 51, 51);font-family:helvetica, arial, freesans, clean, sans-serif;font-size:12px;line-height:18px">sudo dtrace -s route.d -p `pgrep beam.smp`</span></div>

<div><br></div><div>The examples are really basic but I think they illustrate the point. With that Erlang branch you could enable more granular probes than with the usual pid provider or dtruss.</div><div><br></div><div>
BTW, this works on Mac and Solaris. AFAIK on Linux some Kernel modules are required.</div>
<div><br></div><div>Cheers,</div><div><br></div><div>Alvaro</div>
</div></blockquote></div></div><blockquote type="cite"><div><span>_______________________________________________</span><div class="im"><br><span>rabbitmq-discuss mailing list</span><br><span><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank"><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a></a></span><br>
<span><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank"><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a></a></span><br></div></div></blockquote></div></blockquote>
</div><br>
</div></blockquote></body></html>