[rabbitmq-discuss] Regular supervisor shutdown errors?

Sat May 26 05:00:05 BST 2012

I'm seeing somewhat regular (but not in a tight loop) occurrences of these notices in my rabbit@<host>-sasl.log.

Sometimes the reason is listed as "noproc", other times as "shutdown".  Are these normal?  If not, how can I debug?

> =SUPERVISOR REPORT==== 26-May-2012::02:09:34 ===
>      Supervisor: {<0.549.2188>,
>                                            rabbit_channel_sup_sup}
>      Context:    shutdown_error
>      Reason:     noproc
>      Offender:   [{pid,<0.551.2188>},
>                   {name,channel_sup},
>                   {mfa,{rabbit_channel_sup,start_link,[]}},
>                   {restart_type,temporary},
>                   {shutdown,infinity},
>                   {child_type,supervisor}]
> 
> 
> =SUPERVISOR REPORT==== 26-May-2012::02:15:52 ===
>      Supervisor: {<0.31254.2188>,
>                                            rabbit_channel_sup_sup}
>      Context:    shutdown_error
>      Reason:     noproc
>      Offender:   [{pid,<0.31256.2188>},
>                   {name,channel_sup},
>                   {mfa,{rabbit_channel_sup,start_link,[]}},
>                   {restart_type,temporary},
>                   {shutdown,infinity},
>                   {child_type,supervisor}]
> 
> 
> =SUPERVISOR REPORT==== 26-May-2012::02:18:20 ===
>      Supervisor: {<0.15873.2189>,
>                                            rabbit_channel_sup_sup}
>      Context:    shutdown_error
>      Reason:     shutdown
>      Offender:   [{pid,<0.15875.2189>},
>                   {name,channel_sup},
>                   {mfa,{rabbit_channel_sup,start_link,[]}},
>                   {restart_type,temporary},
>                   {shutdown,infinity},
>                   {child_type,supervisor}]
> 
> 
> =SUPERVISOR REPORT==== 26-May-2012::03:52:54 ===
>      Supervisor: {<0.23330.3>,rabbit_channel_sup_sup}
>      Context:    shutdown_error
>      Reason:     shutdown
>      Offender:   [{pid,<0.23340.3>},
>                   {name,channel_sup},
>                   {mfa,{rabbit_channel_sup,start_link,[]}},
>                   {restart_type,temporary},
>                   {shutdown,infinity},
>                   {child_type,supervisor}]
> 
> 
> =SUPERVISOR REPORT==== 26-May-2012::03:53:48 ===
>      Supervisor: {<0.27559.3>,rabbit_channel_sup_sup}
>      Context:    shutdown_error
>      Reason:     shutdown
>      Offender:   [{pid,<0.27561.3>},
>                   {name,channel_sup},
>                   {mfa,{rabbit_channel_sup,start_link,[]}},
>                   {restart_type,temporary},
>                   {shutdown,infinity},
>                   {child_type,supervisor}]
> 
> 
> =SUPERVISOR REPORT==== 26-May-2012::03:54:22 ===
>      Supervisor: {<0.30115.3>,rabbit_channel_sup_sup}
>      Context:    shutdown_error
>      Reason:     shutdown
>      Offender:   [{pid,<0.30117.3>},
>                   {name,channel_sup},
>                   {mfa,{rabbit_channel_sup,start_link,[]}},
>                   {restart_type,temporary},
>                   {shutdown,infinity},
>                   {child_type,supervisor}]

I've also seen 3 of my instances of rabbitmq crash this evening. Haven't changed anything about the configuration or versions or anything in weeks, but clearly something is going on.  One one of the crashed instances, the startup_err log shows me:

> Crash dump was written to: erl_crash.dump
> eheap_alloc: Cannot allocate 2850821240 bytes of memory (of type "old_heap").
> /usr/sbin/rabbitmq-server: line 42: 31512 Aborted                 /usr/lib/rabbitmq/bin/${SCRIPT} "$@"

I can't find the erl_crash.dump file, however.  Where should that live?

Rabbit seems to startup okay, except that when this happens, I lose all my persistent messages in a queue (issue I mentioned on the list last week, never figured it out, I'm moving away from rabbit for my persistent message needs, this is just too scary an issue for my production data at the moment). Not sure if these warnings in the logs are related to the crashes that I've seen, or the lost messages.

-Will

________________
Will Koffel
CTO, Thumb™
51 E 12th St., 4th Floor
New York, NY 10003
Office: (212) 673-8650
Mobile: (617) 575-WILL
@thumb
www.thumb.it

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120526/1625d593/attachment.htm>