<div dir="ltr">Anything I can do to move the ball forward here? This keeps on happening.</div><div class="gmail_extra"><br clear="all"><div>Michael Sander<div><a href="mailto:mes65@cornell.edu" target="_blank">mes65@cornell.edu</a></div>
<div>607-227-9859</div></div>
<br><br><div class="gmail_quote">On Tue, Apr 8, 2014 at 3:49 AM, Michael Sander <span dir="ltr"><<a href="mailto:mes65@cornell.edu" target="_blank">mes65@cornell.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Unfortunately, I had to restart rabbitmq because it's on a production machine. Attached is screenshot after the restart. Notice that this time, free disk space limit is available.<div class=""><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Please let me know what additional information you would like. If it goes down again, I will collect it.</span><br></div></div><div class="gmail_extra">
<br clear="all"><div>Michael Sander<div><br></div></div>
<br><br><div class="gmail_quote"><div><div class="h5">On Tue, Apr 8, 2014 at 3:42 AM, Michael Sander <span dir="ltr"><<a href="mailto:mes65@cornell.edu" target="_blank">mes65@cornell.edu</a>></span> wrote:<br></div>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
<div dir="ltr">Hi Matthias,<div><br></div><div>I just checked rabbit the second after sending that, and it appears to have crashed. Here is some output, that you may find useful. Notice that erlang appears to be alive even though rabbitmq is not.</div>
<div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="courier new, monospace">~$ ps aux|grep rabbit</font></div></div><div><div><font face="courier new, monospace">rabbitmq 1761 0.0 0.0 10836 160 ? S Apr07 0:01 /usr/lib/erlang/erts-5.9.1/bin/epmd -daemon</font></div>
</div><div><div><font face="courier new, monospace">1001 26076 0.0 0.0 6308 600 pts/1 S+ 07:34 0:00 grep rabbit</font></div></div><div><div><font face="courier new, monospace">~$ ps aux|grep erlan</font></div>
</div><div><div><font face="courier new, monospace">rabbitmq 1761 0.0 0.0 10836 160 ? S Apr07 0:01 /usr/lib/erlang/erts-5.9.1/bin/epmd -daemon</font></div></div><div><div><font face="courier new, monospace">1001 26078 0.0 0.0 6308 600 pts/1 S+ 07:34 0:00 grep erlan</font></div>
</div><div><div><font face="courier new, monospace">~$ df -h</font></div></div><div><div><font face="courier new, monospace">Filesystem Size Used Avail Use% Mounted on</font></div>
</div><div><div><font face="courier new, monospace">rootfs 9.9G 6.1G 3.3G 65% /</font></div></div><div><div><font face="courier new, monospace">udev 10M 0 10M 0% /dev</font></div>
</div><div><div><font face="courier new, monospace">tmpfs 181M 128K 181M 1% /run</font></div></div><div><div><font face="courier new, monospace">/dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 9.9G 6.1G 3.3G 65% /</font></div>
</div><div><div><font face="courier new, monospace">tmpfs 5.0M 0 5.0M 0% /run/lock</font></div></div><div><div><font face="courier new, monospace">tmpfs 362M 0 362M 0% /run/shm</font></div>
</div><div><div><font face="courier new, monospace">~$ sudo df -h</font></div></div><div><div><font face="courier new, monospace">Filesystem Size Used Avail Use% Mounted on</font></div>
</div><div><div><font face="courier new, monospace">rootfs 9.9G 6.1G 3.3G 65% /</font></div></div><div><div><font face="courier new, monospace">udev 10M 0 10M 0% /dev</font></div>
</div><div><div><font face="courier new, monospace">tmpfs 181M 128K 181M 1% /run</font></div></div><div><div><font face="courier new, monospace">/dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 9.9G 6.1G 3.3G 65% /</font></div>
</div><div><div><font face="courier new, monospace">tmpfs 5.0M 0 5.0M 0% /run/lock</font></div></div><div><div><font face="courier new, monospace">tmpfs 362M 0 362M 0% /run/shm</font></div>
</div><div><div><font face="courier new, monospace">~$ tail -n 100 /var/log/rabbitmq/rabbit@ocr-proc-2-sasl.log<br></font></div><div><font face="courier new, monospace">~$ tail -n 100 /var/log/rabbitmq/rabbit@ocr-proc-2-sasl.log.1</font></div>
<div><font face="courier new, monospace"> crasher:</font></div><div><font face="courier new, monospace"> initial call: rabbit_disk_monitor:init/1</font></div><div><font face="courier new, monospace"> pid: <0.19499.0></font></div>
<div><font face="courier new, monospace"> registered_name: []</font></div><div><font face="courier new, monospace"> exception exit: unsupported_platform</font></div><div><font face="courier new, monospace"> in function gen_server:init_it/6 (gen_server.erl, line 320)</font></div>
<div><font face="courier new, monospace"> ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>]</font></div><div><font face="courier new, monospace"> messages: []</font></div><div><font face="courier new, monospace"> links: [<0.180.0>]</font></div>
<div><font face="courier new, monospace"> dictionary: []</font></div><div><font face="courier new, monospace"> trap_exit: false</font></div><div><font face="courier new, monospace"> status: running</font></div><div>
<font face="courier new, monospace"> heap_size: 6765</font></div><div><font face="courier new, monospace"> stack_size: 24</font></div><div><font face="courier new, monospace"> reductions: 13592</font></div><div>
<font face="courier new, monospace"> neighbours:</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">=SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 ===</font></div><div><font face="courier new, monospace"> Supervisor: {local,rabbit_disk_monitor_sup}</font></div>
<div><font face="courier new, monospace"> Context: start_error</font></div><div><font face="courier new, monospace"> Reason: unsupported_platform</font></div><div><font face="courier new, monospace"> Offender: [{pid,{restarting,<0.5000.0>}},</font></div>
<div><font face="courier new, monospace"> {name,rabbit_disk_monitor},</font></div><div><font face="courier new, monospace"> {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},</font></div>
<div><font face="courier new, monospace"> {restart_type,transient},</font></div><div><font face="courier new, monospace"> {shutdown,4294967295},</font></div><div><font face="courier new, monospace"> {child_type,worker}]</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">=CRASH REPORT==== 8-Apr-2014::00:38:08 ===</font></div><div><font face="courier new, monospace"> crasher:</font></div>
<div><font face="courier new, monospace"> initial call: rabbit_disk_monitor:init/1</font></div><div><font face="courier new, monospace"> pid: <0.19502.0></font></div><div><font face="courier new, monospace"> registered_name: []</font></div>
<div><font face="courier new, monospace"> exception exit: unsupported_platform</font></div><div><font face="courier new, monospace"> in function gen_server:init_it/6 (gen_server.erl, line 320)</font></div><div><font face="courier new, monospace"> ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>]</font></div>
<div><font face="courier new, monospace"> messages: []</font></div><div><font face="courier new, monospace"> links: [<0.180.0>]</font></div><div><font face="courier new, monospace"> dictionary: []</font></div>
<div><font face="courier new, monospace"> trap_exit: false</font></div><div><font face="courier new, monospace"> status: running</font></div><div><font face="courier new, monospace"> heap_size: 6765</font></div>
<div>
<font face="courier new, monospace"> stack_size: 24</font></div><div><font face="courier new, monospace"> reductions: 13592</font></div><div><font face="courier new, monospace"> neighbours:</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">=SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 ===</font></div><div><font face="courier new, monospace"> Supervisor: {local,rabbit_disk_monitor_sup}</font></div><div>
<font face="courier new, monospace"> Context: start_error</font></div><div><font face="courier new, monospace"> Reason: unsupported_platform</font></div><div><font face="courier new, monospace"> Offender: [{pid,{restarting,<0.5000.0>}},</font></div>
<div><font face="courier new, monospace"> {name,rabbit_disk_monitor},</font></div><div><font face="courier new, monospace"> {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},</font></div>
<div><font face="courier new, monospace"> {restart_type,transient},</font></div><div><font face="courier new, monospace"> {shutdown,4294967295},</font></div><div><font face="courier new, monospace"> {child_type,worker}]</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">=CRASH REPORT==== 8-Apr-2014::00:38:08 ===</font></div><div><font face="courier new, monospace"> crasher:</font></div>
<div><font face="courier new, monospace"> initial call: rabbit_disk_monitor:init/1</font></div><div><font face="courier new, monospace"> pid: <0.19505.0></font></div><div><font face="courier new, monospace"> registered_name: []</font></div>
<div><font face="courier new, monospace"> exception exit: unsupported_platform</font></div><div><font face="courier new, monospace"> in function gen_server:init_it/6 (gen_server.erl, line 320)</font></div><div><font face="courier new, monospace"> ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.157.0>]</font></div>
<div><font face="courier new, monospace"> messages: []</font></div><div><font face="courier new, monospace"> links: [<0.180.0>]</font></div><div><font face="courier new, monospace"> dictionary: []</font></div>
<div><font face="courier new, monospace"> trap_exit: false</font></div><div><font face="courier new, monospace"> status: running</font></div><div><font face="courier new, monospace"> heap_size: 6765</font></div>
<div>
<font face="courier new, monospace"> stack_size: 24</font></div><div><font face="courier new, monospace"> reductions: 13592</font></div><div><font face="courier new, monospace"> neighbours:</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">=SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 ===</font></div><div><font face="courier new, monospace"> Supervisor: {local,rabbit_disk_monitor_sup}</font></div><div>
<font face="courier new, monospace"> Context: start_error</font></div><div><font face="courier new, monospace"> Reason: unsupported_platform</font></div><div><font face="courier new, monospace"> Offender: [{pid,{restarting,<0.5000.0>}},</font></div>
<div><font face="courier new, monospace"> {name,rabbit_disk_monitor},</font></div><div><font face="courier new, monospace"> {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},</font></div>
<div><font face="courier new, monospace"> {restart_type,transient},</font></div><div><font face="courier new, monospace"> {shutdown,4294967295},</font></div><div><font face="courier new, monospace"> {child_type,worker}]</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">=SUPERVISOR REPORT==== 8-Apr-2014::00:38:08 ===</font></div><div>
<font face="courier new, monospace"> Supervisor: {local,rabbit_disk_monitor_sup}</font></div><div><font face="courier new, monospace"> Context: shutdown</font></div><div><font face="courier new, monospace"> Reason: reached_max_restart_intensity</font></div>
<div><font face="courier new, monospace"> Offender: [{pid,{restarting,<0.5000.0>}},</font></div><div><font face="courier new, monospace"> {name,rabbit_disk_monitor},</font></div><div><font face="courier new, monospace"> {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},</font></div>
<div><font face="courier new, monospace"> {restart_type,transient},</font></div><div><font face="courier new, monospace"> {shutdown,4294967295},</font></div><div><font face="courier new, monospace"> {child_type,worker}]</font></div>
</div><div><font face="courier new, monospace"><div>~$ tail -n 100 /var/log/rabbitmq/rabbit@ocr-proc-2.log</div><div>=WARNING REPORT==== 8-Apr-2014::07:29:47 ===</div><div>closing AMQP connection <0.20361.1> (<a href="http://127.0.0.1:48568" target="_blank">127.0.0.1:48568</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=WARNING REPORT==== 8-Apr-2014::07:29:47 ===</div><div>closing AMQP connection <0.20392.1> (<a href="http://127.0.0.1:48586" target="_blank">127.0.0.1:48586</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=WARNING REPORT==== 8-Apr-2014::07:29:49 ===</div><div>closing AMQP connection <0.20401.1> (<a href="http://127.0.0.1:48589" target="_blank">127.0.0.1:48589</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=WARNING REPORT==== 8-Apr-2014::07:29:50 ===</div><div>closing AMQP connection <0.22633.1> (<a href="http://127.0.0.1:50329" target="_blank">127.0.0.1:50329</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=WARNING REPORT==== 8-Apr-2014::07:29:51 ===</div><div>closing AMQP connection <0.16156.1> (<a href="http://127.0.0.1:44692" target="_blank">127.0.0.1:44692</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:11 ===</div><div>accepting AMQP connection <0.22761.1> (<a href="http://127.0.0.1:50370" target="_blank">127.0.0.1:50370</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=WARNING REPORT==== 8-Apr-2014::07:30:11 ===</div><div>closing AMQP connection <0.22608.1> (<a href="http://127.0.0.1:50316" target="_blank">127.0.0.1:50316</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:11 ===</div><div>accepting AMQP connection <0.22774.1> (<a href="http://127.0.0.1:50371" target="_blank">127.0.0.1:50371</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:11 ===</div><div>accepting AMQP connection <0.22777.1> (<a href="http://127.0.0.1:50372" target="_blank">127.0.0.1:50372</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:26 ===</div><div>accepting AMQP connection <0.22796.1> (<a href="http://127.0.0.1:50383" target="_blank">127.0.0.1:50383</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:26 ===</div><div>accepting AMQP connection <0.22805.1> (<a href="http://127.0.0.1:50384" target="_blank">127.0.0.1:50384</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:26 ===</div><div>accepting AMQP connection <0.22810.1> (<a href="http://127.0.0.1:50385" target="_blank">127.0.0.1:50385</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:27 ===</div><div>accepting AMQP connection <0.22825.1> (<a href="http://127.0.0.1:50386" target="_blank">127.0.0.1:50386</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:29 ===</div><div>accepting AMQP connection <0.22834.1> (<a href="http://127.0.0.1:50387" target="_blank">127.0.0.1:50387</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:30 ===</div><div>accepting AMQP connection <0.22843.1> (<a href="http://127.0.0.1:50388" target="_blank">127.0.0.1:50388</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:31 ===</div><div>accepting AMQP connection <0.22852.1> (<a href="http://127.0.0.1:50389" target="_blank">127.0.0.1:50389</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:34 ===</div><div>accepting AMQP connection <0.22863.1> (<a href="http://127.0.0.1:50394" target="_blank">127.0.0.1:50394</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:34 ===</div><div>accepting AMQP connection <0.22866.1> (<a href="http://127.0.0.1:50395" target="_blank">127.0.0.1:50395</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=WARNING REPORT==== 8-Apr-2014::07:30:36 ===</div><div>closing AMQP connection <0.22852.1> (<a href="http://127.0.0.1:50389" target="_blank">127.0.0.1:50389</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:36 ===</div><div>accepting AMQP connection <0.22883.1> (<a href="http://127.0.0.1:50399" target="_blank">127.0.0.1:50399</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=WARNING REPORT==== 8-Apr-2014::07:30:37 ===</div><div>closing AMQP connection <0.22761.1> (<a href="http://127.0.0.1:50370" target="_blank">127.0.0.1:50370</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=WARNING REPORT==== 8-Apr-2014::07:30:38 ===</div><div>closing AMQP connection <0.22796.1> (<a href="http://127.0.0.1:50383" target="_blank">127.0.0.1:50383</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:39 ===</div><div>accepting AMQP connection <0.22893.1> (<a href="http://127.0.0.1:50403" target="_blank">127.0.0.1:50403</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=WARNING REPORT==== 8-Apr-2014::07:30:39 ===</div><div>closing AMQP connection <0.22810.1> (<a href="http://127.0.0.1:50385" target="_blank">127.0.0.1:50385</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:41 ===</div><div>accepting AMQP connection <0.22902.1> (<a href="http://127.0.0.1:50409" target="_blank">127.0.0.1:50409</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:43 ===</div><div>accepting AMQP connection <0.22913.1> (<a href="http://127.0.0.1:50411" target="_blank">127.0.0.1:50411</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:49 ===</div><div>accepting AMQP connection <0.22925.1> (<a href="http://127.0.0.1:50420" target="_blank">127.0.0.1:50420</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:49 ===</div><div>accepting AMQP connection <0.22928.1> (<a href="http://127.0.0.1:50421" target="_blank">127.0.0.1:50421</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><br></div><div>=WARNING REPORT==== 8-Apr-2014::07:30:50 ===</div><div>closing AMQP connection <0.22660.1> (<a href="http://127.0.0.1:50332" target="_blank">127.0.0.1:50332</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>):</div>
<div>connection_closed_abruptly</div><div><br></div><div>=INFO REPORT==== 8-Apr-2014::07:30:51 ===</div><div>accepting AMQP connection <0.22945.1> (<a href="http://127.0.0.1:50423" target="_blank">127.0.0.1:50423</a> -> <a href="http://127.0.0.1:5672" target="_blank">127.0.0.1:5672</a>)</div>
<div><div>~$ tail -n 100 /var/log/rabbitmq/shutdown_err</div><div>/usr/lib/rabbitmq/bin/rabbitmqctl: 1: /etc/rabbitmq/rabbitmq-env.conf: ocr-proc-2=rabbit@localhost: not found</div><div>~$ tail -n 100 /var/log/rabbitmq/shutdown_log</div>
<div>Stopping and halting node 'rabbit@ocr-proc-2' ...</div><div>...done.</div></div><div><div>~$ tail -n 100 /var/log/rabbitmq/startup_err</div><div>/usr/lib/rabbitmq/bin/rabbitmq-server: 1: /etc/rabbitmq/rabbitmq-env.conf: ocr-proc-2=rabbit@localhost: not found</div>
<div>Killed</div><div>~$ tail -n 100 /var/log/rabbitmq/startup_log</div><div><br></div><div> RabbitMQ 3.2.4. Copyright (C) 2007-2013 GoPivotal, Inc.</div><div> ## ## Licensed under the MPL. See <a href="http://www.rabbitmq.com/" target="_blank">http://www.rabbitmq.com/</a></div>
<div> ## ##</div><div> ########## Logs: /var/log/rabbitmq/rabbit@ocr-proc-2.log</div><div> ###### ## /var/log/rabbitmq/rabbit@ocr-proc-2-sasl.log</div><div> ##########</div><div> Starting broker... completed with 6 plugins.</div>
</div></font></div><div><br></div></blockquote><div><br></div><div><br></div>
<span><font color="#888888">
</font></span></div></div></div><div class="gmail_extra"><span><font color="#888888"><br clear="all"><div>Michael Sander<div><a href="mailto:mes65@cornell.edu" target="_blank">mes65@cornell.edu</a></div><div><a href="tel:607-227-9859" value="+16072279859" target="_blank">607-227-9859</a></div>
</div></font></span><div><div class="h5"><div><div>
<br><br><div class="gmail_quote">On Tue, Apr 8, 2014 at 3:33 AM, Michael Sander <span dir="ltr"><<a href="mailto:mes65@cornell.edu" target="_blank">mes65@cornell.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><span style="font-family:arial,sans-serif;font-size:12.571428298950195px">Hi Matthais,</span><div style="font-family:arial,sans-serif;font-size:12.571428298950195px"><br></div><div style="font-family:arial,sans-serif;font-size:12.571428298950195px">
What I sent you was everything I had. However, I did check ps -aux after the crash and rabbitmq-server was definitely not in there. I will turn off the cron jobs that automatically restart rabbitmq, and I'll let you know if I see it again.</div>
<div style="font-family:arial,sans-serif;font-size:12.571428298950195px"><br></div></div><div style="font-family:arial,sans-serif;font-size:12.571428298950195px"><div>Here is the output of the command.</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div><font face="courier new, monospace">$ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP /var/lib/rabbitmq/mnes</font></div><div><font face="courier new, monospace">ia/").'</font></div>
<div><font face="courier new, monospace">"Filesystem 1024-blocks Used Available Capacity Mounted on\n/dev/disk/by-uuid/36fd30d4-ea87-419f-a6a4-a1a3cf290ff1 10320184 6348300 3447648 65% /\n"</font></div>
<div><font face="courier new, monospace">...done.</font></div></blockquote></div><div class="gmail_extra"><br></div><div class="gmail_extra">Also, I'm not sure whether it will help you, but attached is a screenshot of the rabbitmq console. If you see the start of the top chart at 19:00, there is a sharp increase in the queued messages. That's when I restarted rabbitmq after the crash. Everything before that was flat. Another point to note is that it currently says that the disk space is unavailable. I definitely remember seeing a value there at some point before, I don't know what causes that to occur. <div>
<div>
<br></div><div>I've turned off my rabbimq auto-start cron jobs, I'll let you know if I see the crash again.<div><br></div><div>Thanks again.</div><div><br></div><div><span style="font-family:arial,sans-serif;font-size:12.571428298950195px">Best,</span></div>
</div></div></div><div class="gmail_extra"><span><font color="#888888"><br clear="all"><div>Michael Sander</div></font></span><div><div><div><br></div><div><br></div><br><div class="gmail_quote">
On Tue, Apr 8, 2014 at 1:18 AM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Michael,<div><br>
<br>
On 08/04/14 02:50, Michael Sander wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Full logs are attached. You'll notice that it crashes pretty often now.<br>
</blockquote>
<br></div>
The disk_monitor is crashing frequently, yes, but in none of the instances in the logs that actually took down rabbit (notice that there are no rabbit starts recorded in the rabbit.log); the disk_monitor restarts just fine and the bunny lives on.<br>
<br>
Do you have the logs covering the time period around the crash?<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Here are the output of the commands<br>
<br>
$ sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP<br>
/var/lib/rabbitmq/mnesia/")'<br>
Error: syntax error before:<br>
</blockquote>
<br></div>
Ah, sorry, missed a full stop. Should be<br>
<br>
sudo rabbitmqctl eval 'rabbit_misc:os_cmd("/bin/df -kP /var/lib/rabbitmq/mnesia/").'<span><font color="#888888"><br>
<br>
<br>
Matthias.<br>
</font></span></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div></div>
</blockquote></div><br></div>