I am not quite sure on the function evaluation order, but it might help to know that <0.159.0> is the disk_log process:<br><br><0.159.0>������������ disk_log:init/2���������������������� 1597�� 879997��� 0<br>��������������������� disk_log:loop/1������������������������� 4������������� <br>
<br>_steve<br><br><div class="gmail_quote">On Thu, Oct 22, 2009 at 5:47 PM, Stephen Day <span dir="ltr"><<a href="mailto:sjaday@gmail.com">sjaday@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I won't bore you with all the output, but I tracked down the binary usage to these two processes:<br><br>[{Pid1, _Info, _Bin}, {Pid2, _Info2, _Bin2} | Other ] = [{P,
process_info(P), BinInfo} || {P, {binary, BinInfo}} <- [{P,
process_info(P, binary)} || P <- processes()], length(BinInfo) >
100000].<br><br><0.157.0>������������ gen:init_it/6��������������������� 1682835� 1873131��� 0<br>��������������������� gen_server2:process_next_msg/8��������� 13������������� <br><0.158.0>������������ rabbit_persister:init/1���������� 19590700 29789397��� 0<br>
rabbit_persister����� gen_server2:process_next_msg/8��������� 13������������� <br><br>I tried your suggestion to free memory and check, but it looks most was held up in the persister:<br><br>35> M = [{erlang:garbage_collect(P), memory(total)} || P <- erlang:processes()].<br>
<br>51> [{P,Mem} || {Mem, P} <- lists:zip( [Me||{true, Me} <- M], processes()), Mem < 842757048].<br>[{<0.148.0>,842753448},<br>�{<0.149.0>,842700248},<br>�{<0.150.0>,842700248},<br>�{<0.151.0>,842700248},<br>
�{<0.152.0>,842697224},<br>�{<0.154.0>,842697792},<br>�{<0.155.0>,842724104},<br>�{<0.156.0>,842712824},<br>�{<0.157.0>,825951032},<br>�{<0.158.0>,602886872},<br>�{<0.159.0>,345002144},<br>
�{<0.177.0>,345002144},<br>�{<0.178.0>,345002144},<br>�{<0.179.0>,345002144},<br>�{<0.180.0>,345002144},<br>�{<0.181.0>,345002144},<br>�{<0.182.0>,345002144},<br>�{<0.183.0>,345002144},<br>
�{<0.184.0>,345002144},<br>�{<0.245.0>,345000624},<br>�{<0.247.0>,345001520},<br>�{<0.248.0>,344996984},<br>�{<0.249.0>,344995464},<br>�{<0.250.0>,344995512},<br>�{<0.252.0>,344996416},<br>
�{<0.253.0>,344991880},<br>�{<0.254.0>,344991928},<br>�{<0.261.0>,...},<br>�{...}|...]<br><br>So it looks like the large chunks are held up between between gen_server2 and rabbit_persister.<br><br>_steve<div>
<div></div><div class="h5"><br>
<br><div class="gmail_quote">On Thu, Oct 22, 2009 at 4:24 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@lshift.net" target="_blank">matthias@lshift.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Stephen,<div><br>
<br>
Stephen Day wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
(rabbit@vs-dfw-ctl11)5> [erlang:garbage_collect(P) || P <- erlang:processes()].<br>
[true,true,true,true,true,true,true,true,true,true,true,<br>
�true,true,true,true,true,true,true,true,true,true,true,true,<br>
�true,true,true,true,true,true|...]<br>
<br>
(rabbit@vs-dfw-ctl11)6> memory(). � � � � � � � � � � � � � � � � � � � � � � [{total,145833144},<br>
�{processes,50900752},<br>
�{processes_used,50896864},<br>
�{system,94932392},<br>
�{atom,514765},<br>
�{atom_used,488348},<br>
�{binary,24622512},<br>
�{code,3880064},<br>
�{ets,64745716}]<br>
<br>
This really cut down on usage, so its likely that the binary gc is falling behind rabbits requirements.<br>
</blockquote>
<br></div>
Agreed.<div><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
How do I track down the uncollected binary heap usage to a process?<br>
</blockquote>
<br></div>
Binaries are shared between processes and ref counted, so no single process owns them. There is a process_info item called 'binary' that provides information on the binaries referenced by a process, but I've never looked at that myself, so don't know how useful the contained info is.<br>
<br>
One thing you could try is to run the above garbage_collect code interleaved with the memory reporting code to identify which process results in the biggest drop in memory memory usage when gc'ed.<br>
<br>
<br>
Regards,<br><font color="#888888">
<br>
Matthias.<br>
</font></blockquote></div><br>
</div></div></blockquote></div><br>