I am not quite sure on the function evaluation order, but it might help to know that <0.159.0> is the disk_log process:<br><br><0.159.0> disk_log:init/2 1597 879997 0<br> disk_log:loop/1 4 <br>
<br>_steve<br><br><div class="gmail_quote">On Thu, Oct 22, 2009 at 5:47 PM, Stephen Day <span dir="ltr"><<a href="mailto:sjaday@gmail.com">sjaday@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I won't bore you with all the output, but I tracked down the binary usage to these two processes:<br><br>[{Pid1, _Info, _Bin}, {Pid2, _Info2, _Bin2} | Other ] = [{P,
process_info(P), BinInfo} || {P, {binary, BinInfo}} <- [{P,
process_info(P, binary)} || P <- processes()], length(BinInfo) >
100000].<br><br><0.157.0> gen:init_it/6 1682835 1873131 0<br> gen_server2:process_next_msg/8 13 <br><0.158.0> rabbit_persister:init/1 19590700 29789397 0<br>
rabbit_persister gen_server2:process_next_msg/8 13 <br><br>I tried your suggestion to free memory and check, but it looks most was held up in the persister:<br><br>35> M = [{erlang:garbage_collect(P), memory(total)} || P <- erlang:processes()].<br>
<br>51> [{P,Mem} || {Mem, P} <- lists:zip( [Me||{true, Me} <- M], processes()), Mem < 842757048].<br>[{<0.148.0>,842753448},<br> {<0.149.0>,842700248},<br> {<0.150.0>,842700248},<br> {<0.151.0>,842700248},<br>
{<0.152.0>,842697224},<br> {<0.154.0>,842697792},<br> {<0.155.0>,842724104},<br> {<0.156.0>,842712824},<br> {<0.157.0>,825951032},<br> {<0.158.0>,602886872},<br> {<0.159.0>,345002144},<br>
{<0.177.0>,345002144},<br> {<0.178.0>,345002144},<br> {<0.179.0>,345002144},<br> {<0.180.0>,345002144},<br> {<0.181.0>,345002144},<br> {<0.182.0>,345002144},<br> {<0.183.0>,345002144},<br>
{<0.184.0>,345002144},<br> {<0.245.0>,345000624},<br> {<0.247.0>,345001520},<br> {<0.248.0>,344996984},<br> {<0.249.0>,344995464},<br> {<0.250.0>,344995512},<br> {<0.252.0>,344996416},<br>
{<0.253.0>,344991880},<br> {<0.254.0>,344991928},<br> {<0.261.0>,...},<br> {...}|...]<br><br>So it looks like the large chunks are held up between between gen_server2 and rabbit_persister.<br><br>_steve<div>
<div></div><div class="h5"><br>
<br><div class="gmail_quote">On Thu, Oct 22, 2009 at 4:24 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@lshift.net" target="_blank">matthias@lshift.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Stephen,<div><br>
<br>
Stephen Day wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
(rabbit@vs-dfw-ctl11)5> [erlang:garbage_collect(P) || P <- erlang:processes()].<br>
[true,true,true,true,true,true,true,true,true,true,true,<br>
true,true,true,true,true,true,true,true,true,true,true,true,<br>
true,true,true,true,true,true|...]<br>
<br>
(rabbit@vs-dfw-ctl11)6> memory(). [{total,145833144},<br>
{processes,50900752},<br>
{processes_used,50896864},<br>
{system,94932392},<br>
{atom,514765},<br>
{atom_used,488348},<br>
{binary,24622512},<br>
{code,3880064},<br>
{ets,64745716}]<br>
<br>
This really cut down on usage, so its likely that the binary gc is falling behind rabbits requirements.<br>
</blockquote>
<br></div>
Agreed.<div><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
How do I track down the uncollected binary heap usage to a process?<br>
</blockquote>
<br></div>
Binaries are shared between processes and ref counted, so no single process owns them. There is a process_info item called 'binary' that provides information on the binaries referenced by a process, but I've never looked at that myself, so don't know how useful the contained info is.<br>
<br>
One thing you could try is to run the above garbage_collect code interleaved with the memory reporting code to identify which process results in the biggest drop in memory memory usage when gc'ed.<br>
<br>
<br>
Regards,<br><font color="#888888">
<br>
Matthias.<br>
</font></blockquote></div><br>
</div></div></blockquote></div><br>