[rabbitmq-discuss] rabbitmq_status_web crashed on daylight savings time switchover, took down RabbitMQ

[Greg Campbell]

Here are the remainder of messages in the rabbit.log.  These are the last
messages until I manually restarted the process after noting that it had
crashed.

=ERROR REPORT==== 13-Mar-2010::18:00:08 ===
{mochiweb_socket_server,235,
    {child_error,
        {{{case_clause,[]},
          [{httpd_util,rfc1123_date,1},
           {rabbit_status_web,internal_update,1},
           {rabbit_status_web,handle_call,3},
           {gen_server,handle_msg,5},
           {proc_lib,init_p_do_apply,3}]},
         {gen_server2,call,[rabbit_status_web,get_context]}}}}

=ERROR REPORT==== 13-Mar-2010::18:00:08 ===
{mochiweb_socket_server,235,
    {child_error,
        {{{case_clause,[]},
          [{httpd_util,rfc1123_date,1},
           {rabbit_status_web,internal_update,1},
           {rabbit_status_web,init,1},
           {gen_server,init_it,6},
           {proc_lib,init_p_do_apply,3}]},
         {gen_server2,call,[rabbit_status_web,get_context]}}}}

=INFO REPORT==== 13-Mar-2010::18:00:08 ===
    application: rabbit_status
    exited: shutdown
    type: permanent

=INFO REPORT==== 13-Mar-2010::18:00:08 ===
stopped TCP Listener on 0.0.0.0:5672

=INFO REPORT==== 13-Mar-2010::18:00:08 ===
closing TCP connection <0.1242.0> from 10.8.25.1:59325

=INFO REPORT==== 13-Mar-2010::18:00:08 ===
closing TCP connection <0.30625.41> from 10.8.26.2:58929

=INFO REPORT==== 13-Mar-2010::18:00:08 ===
closing TCP connection <0.24377.64> from 192.168.5.50:3316

=INFO REPORT==== 13-Mar-2010::18:00:08 ===
closing TCP connection <0.27424.60> from 10.8.24.1:34061

=INFO REPORT==== 13-Mar-2010::18:00:08 ===
closing TCP connection <0.30634.41> from 10.8.26.1:53751



On Mon, Mar 15, 2010 at 9:29 AM, Matthew Sackman <matthew at lshift.net> wrote:

> Hi Greg,
>
> On Sun, Mar 14, 2010 at 07:45:35PM -0700, Greg Campbell wrote:
> > ** Generic server rabbit_status_web terminating
> > ** Last message in was get_context
> > ** When Server state == {state,1268531993140,"Sun, 14 Mar 2010 09:59:53
> > GMT",
> >
> > ... a bunch of queue status data...
> >
> > ** Reason for termination ==
> > ** {{case_clause,[]},
> >     [{httpd_util,rfc1123_date,1},
> >      {rabbit_status_web,internal_update,1},
> >      {rabbit_status_web,handle_call,3},
> >      {gen_server,handle_msg,5},
> >      {proc_lib,init_p_do_apply,3}]}
>
> This doesn't indicate that Rabbit itself crashed, only that the
> rabbit_status_web process crashed. That process is under a supervisor
> hierarchy and should have been automatically restarted.
>
> I have reproduced this (in that I can get a very similar crash to happen,
> though without messing with my clock), and whilst similar entries in the
> logs appear, the rabbit_status_web process restarts correctly, and
> Rabbit itself continues on, perfectly happily.
>
> > The underlying issue in httpd_util:rfc1123_date appears to be an Erlang
> bug,
> > which I believe has been fixed in R13B04 (we're still running R13B03 on
> the
> > server, though):
> >
> http://www.erlang.org/cgi-bin/ezmlm-cgi?2:mss:1681:201001:jffdfifdokimdicjnpcp
> > .
>
> Interesting bug, and I'm glad it's now been fixed.
>
> > However, it might be worth ensuring that a crash in a plugin doesn't
> cause
> > the entire system to shut down.
>
> We don't think it did ;) Are there any more entries in the logs that you
> have that indicate anything else went wrong? It's possible the same bug
> affected more than just the plugin, but certainly the entry you showed
> us just shows a process within the status plugin going down which should
> have been immediately restarted.
>
> Matthew
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100315/0245327e/attachment.htm