<div dir="ltr">Hi Matthias,<div><br></div><div>Full logs are attached. You'll notice that it crashes pretty often now.</div><div><br></div><div>Here are the output of the commands</div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><div><font face="courier new, monospace">$ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP /var/lib/rabbitmq/mnesia/")'</font></div></div><div><div><font face="courier new, monospace">Error: syntax error before:</font></div>
</div><div><div><font face="courier new, monospace">$ /bin/df -kP /var/lib/rabbitmq/mnesia/</font></div></div><div><div><font face="courier new, monospace">Filesystem 1024-blocks Used Available Capacity Mounted on</font></div>
</div><div><div><font face="courier new, monospace">/dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 10320184 6358284 3437664 65% /</font></div></div></blockquote><div><br></div><div>I just noticed the following in the non sasl log, I should have included it in the original:</div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="courier new, monospace">=ERROR REPORT==== 7-Apr-2014::10:08:34 ===</font></div></div><div><div><font face="courier new, monospace">** Generic server rabbit_disk_monitor terminating </font></div>
</div><div><div><font face="courier new, monospace">** Last message in was update</font></div></div><div><div><font face="courier new, monospace">** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@ocr-proc-2",</font></div>
</div><div><div><font face="courier new, monospace"> 50000000,3702104064,100,10000,</font></div></div><div><div><font face="courier new, monospace"> #Ref<0.0.1.41329>,false}</font></div>
</div><div><div><font face="courier new, monospace">** Reason for termination == </font></div></div><div><div><font face="courier new, monospace">** {{badmatch,[]},</font></div></div><div><div><font face="courier new, monospace"> [{rabbit_disk_monitor,parse_free_unix,1,[]},</font></div>
</div><div><div><font face="courier new, monospace"> {rabbit_disk_monitor,internal_update,1,[]},</font></div></div><div><div><font face="courier new, monospace"> {rabbit_disk_monitor,handle_info,2,[]},</font></div>
</div><div><div><font face="courier new, monospace"> {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},</font></div></div><div><div><font face="courier new, monospace"> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}</font></div>
</div><div><div><font face="courier new, monospace"><br></font></div></div><div><div><font face="courier new, monospace">=ERROR REPORT==== 7-Apr-2014::10:08:35 ===</font></div></div><div><div><font face="courier new, monospace">** Generic server rabbit_mgmt_external_stats terminating </font></div>
</div><div><div><font face="courier new, monospace">** Last message in was emit_update</font></div></div><div><div><font face="courier new, monospace">** When Server state == {state,32768}</font></div></div><div><div><font face="courier new, monospace">** Reason for termination == </font></div>
</div><div><div><font face="courier new, monospace">** {{{badmatch,[]},</font></div></div><div><div><font face="courier new, monospace"> [{rabbit_disk_monitor,parse_free_unix,1,[]},</font></div></div><div><div><font face="courier new, monospace"> {rabbit_disk_monitor,internal_update,1,[]},</font></div>
</div><div><div><font face="courier new, monospace"> {rabbit_disk_monitor,handle_info,2,[]},</font></div></div><div><div><font face="courier new, monospace"> {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},</font></div>
</div><div><div><font face="courier new, monospace"> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},</font></div></div><div><div><font face="courier new, monospace"> {gen_server,call,[rabbit_disk_monitor,get_disk_free_limit,infinity]}}</font></div>
</div></blockquote><div><br></div></div><div class="gmail_extra"><br clear="all"><div>Michael Sander<div><a href="mailto:mes65@cornell.edu" target="_blank">mes65@cornell.edu</a></div><div>607-227-9859</div></div>
<br><br><div class="gmail_quote">On Mon, Apr 7, 2014 at 5:29 PM, Michael Sander <span dir="ltr"><<a href="mailto:mes65@cornell.edu" target="_blank">mes65@cornell.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Wow, I had completely forgotten about that, I apologies for the duplicate. It also looks like I dropped the ball on answering your questions. Again my apologies, I was very busy at that time. <div><br></div>
<div>In a few hours I will send you:<div>- The complete sasl and normal logs</div><div>- The output of the command rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP /var/lib/rabbitmq/mnesia/")</div>
<div><br></div></div><div>Best,</div><div class="gmail_extra"><br clear="all"><div>Michael Sander<br></div><div><div class="h5">
<br><br><div class="gmail_quote">On Mon, Apr 7, 2014 at 5:23 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Michael,<div><br>
<br>
On 07/04/14 22:06, Michael Sander wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
I am using a relatively memory-limited machine (n1-highcpu-2 on google<br>
compute engine, details here<br></div>
<<a href="https://developers.google.com/compute/docs/machine-types#highcpu" target="_blank">https://developers.google.<u></u>com/compute/docs/machine-<u></u>types#highcpu</a>>), and<div><br>
there's a possibility that I ran out of memory at some point, but I did<br>
not see any direct evidence of that.<br>
</div></blockquote>
<br>
Ah! Your original report did not mention that this is on GCE. Looking at our bug tracker, I see now that you reported the same issue previously, back in October: <a href="http://rabbitmq.1065348.n5.nabble.com/rabbitmq-nodedown-nodedown-Generic-server-rabbit-disk-monitor-terminating-td30939.html" target="_blank">http://rabbitmq.1065348.n5.<u></u>nabble.com/rabbitmq-nodedown-<u></u>nodedown-Generic-server-<u></u>rabbit-disk-monitor-<u></u>terminating-td30939.html</a>.<div>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The logs I sent were just snippets.<br>
</blockquote>
<br></div>
The complete logs should hopefully shed some further light on the root cause.<span><font color="#888888"><br>
<br>
Matthias.<br>
</font></span></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>