<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I'm wondering these could be running out of memory. I do see them taking very different amounts of RAM on different machines across my stack. It's almost as if the large queues that I have (ones in which messages sit for a while before expiring via their TTL) are getting saved in RAM instead of persistent. But if that were the case, then restarting RabbitMQ would flush them, I presume. And I can definitely restart with the messages still there, which is what made my confident they were durable and persistent.<div><br></div><div>For people familiar with the erl_crash.dump, is my example below a simple "out of memory" error that caused the crash? This is running on EC2, which has no swap space by default, maybe there's a hard crash case here?<br><div><br></div><div>-Will</div><div><br><div><div>On May 26, 2012, at 12:00 AM, Will Koffel wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I'm seeing somewhat regular (but not in a tight loop) occurrences of these notices in my rabbit@<host>-sasl.log.<div><br></div><div>Sometimes the reason is listed as "noproc", other times as "shutdown". Are these normal? If not, how can I debug?</div><div><br></div><div><br></div><div><div><div></div><blockquote type="cite"><div>=SUPERVISOR REPORT==== 26-May-2012::02:09:34 ===</div><div> Supervisor: {<0.549.2188>,</div><div> rabbit_channel_sup_sup}</div><div> Context: shutdown_error</div><div> Reason: noproc</div><div> Offender: [{pid,<0.551.2188>},</div><div> {name,channel_sup},</div><div> {mfa,{rabbit_channel_sup,start_link,[]}},</div><div> {restart_type,temporary},</div><div> {shutdown,infinity},</div><div> {child_type,supervisor}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 26-May-2012::02:15:52 ===</div><div> Supervisor: {<0.31254.2188>,</div><div> rabbit_channel_sup_sup}</div><div> Context: shutdown_error</div><div> Reason: noproc</div><div> Offender: [{pid,<0.31256.2188>},</div><div> {name,channel_sup},</div><div> {mfa,{rabbit_channel_sup,start_link,[]}},</div><div> {restart_type,temporary},</div><div> {shutdown,infinity},</div><div> {child_type,supervisor}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 26-May-2012::02:18:20 ===</div><div> Supervisor: {<0.15873.2189>,</div><div> rabbit_channel_sup_sup}</div><div> Context: shutdown_error</div><div> Reason: shutdown</div><div> Offender: [{pid,<0.15875.2189>},</div><div> {name,channel_sup},</div><div> {mfa,{rabbit_channel_sup,start_link,[]}},</div><div> {restart_type,temporary},</div><div> {shutdown,infinity},</div><div> {child_type,supervisor}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 26-May-2012::03:52:54 ===</div><div> Supervisor: {<0.23330.3>,rabbit_channel_sup_sup}</div><div> Context: shutdown_error</div><div> Reason: shutdown</div><div> Offender: [{pid,<0.23340.3>},</div><div> {name,channel_sup},</div><div> {mfa,{rabbit_channel_sup,start_link,[]}},</div><div> {restart_type,temporary},</div><div> {shutdown,infinity},</div><div> {child_type,supervisor}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 26-May-2012::03:53:48 ===</div><div> Supervisor: {<0.27559.3>,rabbit_channel_sup_sup}</div><div> Context: shutdown_error</div><div> Reason: shutdown</div><div> Offender: [{pid,<0.27561.3>},</div><div> {name,channel_sup},</div><div> {mfa,{rabbit_channel_sup,start_link,[]}},</div><div> {restart_type,temporary},</div><div> {shutdown,infinity},</div><div> {child_type,supervisor}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 26-May-2012::03:54:22 ===</div><div> Supervisor: {<0.30115.3>,rabbit_channel_sup_sup}</div><div> Context: shutdown_error</div><div> Reason: shutdown</div><div> Offender: [{pid,<0.30117.3>},</div><div> {name,channel_sup},</div><div> {mfa,{rabbit_channel_sup,start_link,[]}},</div><div> {restart_type,temporary},</div><div> {shutdown,infinity},</div><div> {child_type,supervisor}]</div></blockquote><div><br></div><div><br></div><div><div>I've also seen 3 of my instances of rabbitmq crash this evening. Haven't changed anything about the configuration or versions or anything in weeks, but clearly something is going on. One one of the crashed instances, the startup_err log shows me:</div><div><br></div><div><div><br></div><div></div></div></div><blockquote type="cite"><div><div><div>Crash dump was written to: erl_crash.dump</div><div>eheap_alloc: Cannot allocate 2850821240 bytes of memory (of type "old_heap").</div><div>/usr/sbin/rabbitmq-server: line 42: 31512 Aborted /usr/lib/rabbitmq/bin/${SCRIPT} "$@"</div></div><div></div></div><div><div apple-content-edited="true">
</div></div></blockquote><div><div apple-content-edited="true"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><div><br></div><div><br></div><div>I can't find the erl_crash.dump file, however. Where should that live?</div><div><br></div><div>Rabbit seems to startup okay, except that when this happens, I lose all my persistent messages in a queue (issue I mentioned on the list last week, never figured it out, I'm moving away from rabbit for my persistent message needs, this is just too scary an issue for my production data at the moment). Not sure if these warnings in the logs are related to the crashes that I've seen, or the lost messages.</div><div><br></div><div>-Will</div><div><br></div><div><br></div><div>________________</div><div>Will Koffel</div><div>CTO, Thumb™</div><div>51 E 12th St., 4th Floor</div><div>New York, NY 10003</div><div>Office: (212) 673-8650</div><div>Mobile: (617) 575-WILL</div><div>@thumb</div><div><a href="http://www.thumb.it/">www.thumb.it</a></div></div></div><div><br></div></div></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br></div></div></div></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><div>________________</div><div>Will Koffel</div><div>CTO, Thumb™</div><div>51 E 12th St., 4th Floor</div><div>New York, NY 10003</div><div>Office: (212) 673-8650</div><div>Mobile: (617) 575-WILL</div><div>@thumb</div><div><a href="http://www.thumb.it/">www.thumb.it</a></div></div></div><div><br></div></div></span></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br></div></div></body></html>