[rabbitmq-discuss] Strange Management Plugin Error

Mon Feb 7 11:56:46 GMT 2011

Hi Guys,

I don't think this has been covered in the mailing list so far (apologies if it has, and I'll go squirrel further, hopefully it's a problem unique to use over here).

We're experiencing a strange problem with the Management Plugin. We quite regularly are greeted with a rather intimidating Erlang stack trace when loading pages of the plugin.

The output we're getting posted at the bottom of this message. Normally hitting refresh a few times seems to clear the message and let us at the interface, other than that our cluster seems to be operating ok.

I've posted below the error as it appears on the page, some entries from the logs, and the output of rabbitmqctl status.

This is the only real problem we've experienced so far with RabbitMQ, we've been very impressed, but if anyone's got any suggestions for fixing up this problem we'll be one step closer to perfection ;-)

Many thanks in advance, if you need any further debug/info I'm more than happy to provide it. :-)

Chris H

And now for the gory details:

>From the browser:

Got response code 500 with body

Internal Server Error
The server encountered an error while processing this request:

{error,{exit,{{undef,[{erlang,max,[0,1297079069756]},
                      {rabbit_mgmt_db,gs_update_add,3},
                      {rabbit_mgmt_db,'-gs_update/2-lc$^1/1-1-',3},
                      {rabbit_mgmt_db,'-gs_update/2-lc$^1/1-1-',3},
                      {lists,foldl,3},
                      {rabbit_mgmt_db,handle_call,3},
                      {gen_server,handle_msg,5},
                      {proc_lib,init_p_do_apply,3}]},
              {gen_server,call,
                          [{global,rabbit_mgmt_db},
                           {get_overview,all},
                           infinity]}},
             [{gen_server,call,3},
              {rabbit_mgmt_db,safe_call,2},
              {rabbit_mgmt_wm_overview,to_json,2},
              {webmachine_resource,resource_call,3},
              {webmachine_resource,do,3},
              {webmachine_decision_core,resource_call,1},
              {webmachine_decision_core,decision,1},
              {webmachine_decision_core,handle_request,2}]}}

>From rabbit\@site1-sasl.log

=CRASH REPORT==== 7-Feb-2011::11:44:29 ===
  crasher:
    pid: <0.9277.47>
    registered_name: []
    exception exit: {undef,[{erlang,max,[0,1297079069756]},
                            {rabbit_mgmt_db,gs_update_add,3},
                            {rabbit_mgmt_db,'-gs_update/2-lc$^1/1-1-',3},
                            {rabbit_mgmt_db,'-gs_update/2-lc$^1/1-1-',3},
                            {lists,foldl,3},
                            {rabbit_mgmt_db,handle_call,3},
                            {gen_server,handle_msg,5},
                            {proc_lib,init_p_do_apply,3}]}
      in function  gen_server:terminate/6
    initial call: rabbit_mgmt_db:init/1
    ancestors: [<0.200.0>,rabbit_sup,<0.111.0>]
    messages: []
    links: [<0.200.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 2584
    stack_size: 23
    reductions: 242620
  neighbours:

=SUPERVISOR REPORT==== 7-Feb-2011::11:44:29 ===
     Supervisor: {global,rabbit_mgmt_global_sup}
     Context:    child_terminated
     Reason:     {undef,[{erlang,max,[0,1297079069756]},
                         {rabbit_mgmt_db,gs_update_add,3},
                         {rabbit_mgmt_db,'-gs_update/2-lc$^1/1-1-',3},
                         {rabbit_mgmt_db,'-gs_update/2-lc$^1/1-1-',3},
                         {lists,foldl,3},
                         {rabbit_mgmt_db,handle_call,3},
                         {gen_server,handle_msg,5},
                         {proc_lib,init_p_do_apply,3}]}
     Offender:   [{pid,<0.9277.47>},
                  {name,rabbit_mgmt_db},
                  {mfa,{rabbit_mgmt_db,start_link,[]}},
                  {restart_type,permanent},
                  {shutdown,5000},
                  {child_type,worker}]

>From rabbit\@site1.log

=ERROR REPORT==== 7-Feb-2011::11:44:29 ===
** Generic server rabbit_mgmt_db terminating
** Last message in was {get_overview,all}
** When Server state == {state,[{channel_exchange_stats,15294516},
                                {channel_queue_exchange_stats,15298613},
                                {channel_queue_stats,15290419},
                                {channel_stats,15282225},
                                {connection_stats,15278128},
                                {consumers,15286322},
                                {queue_stats,15274029}]}
** Reason for termination ==
** {'function not exported',[{erlang,max,[0,1297079069756]},
                             {rabbit_mgmt_db,gs_update_add,3},
                             {rabbit_mgmt_db,'-gs_update/2-lc$^1/1-1-',3},
                             {rabbit_mgmt_db,'-gs_update/2-lc$^1/1-1-',3},
                             {lists,foldl,3},
                             {rabbit_mgmt_db,handle_call,3},
                             {gen_server,handle_msg,5},
                             {proc_lib,init_p_do_apply,3}]}

=ERROR REPORT==== 7-Feb-2011::11:44:29 ===
webmachine error: path="/api/overview"
{error,{exit,{{undef,[{erlang,max,[0,1297079069756]},
                      {rabbit_mgmt_db,gs_update_add,3},
                      {rabbit_mgmt_db,'-gs_update/2-lc$^1/1-1-',3},
                      {rabbit_mgmt_db,'-gs_update/2-lc$^1/1-1-',3},
                      {lists,foldl,3},
                      {rabbit_mgmt_db,handle_call,3},
                      {gen_server,handle_msg,5},
                      {proc_lib,init_p_do_apply,3}]},
              {gen_server,call,
                          [{global,rabbit_mgmt_db},
                           {get_overview,all},
                           infinity]}},
             [{gen_server,call,3},
              {rabbit_mgmt_db,safe_call,2},
              {rabbit_mgmt_wm_overview,to_json,2},
              {webmachine_resource,resource_call,3},
              {webmachine_resource,do,3},
              {webmachine_decision_core,resource_call,1},
              {webmachine_decision_core,decision,1},
              {webmachine_decision_core,handle_request,2}]}}

And finally rabbitmq status so you can see what we're running:

Status of node 'rabbit at site1' ...
[{running_applications,
     [{rabbit_jsonrpc_channel,"RabbitMQ JSON-RPC Channels","2.3.1"},
      {rabbit_management,"RabbitMQ Management Console","2.3.1"},
      {webmachine,"webmachine","1.7.0"},
      {rabbit_management_agent,"RabbitMQ Management Agent","2.3.1"},
      {rabbit_jsonrpc_channel_test,"rabbit_jsonrpc_channel_test","0.01"},
      {amqp_client,"RabbitMQ AMQP Client","2.3.1"},
      {rfc4627_jsonrpc,"JSON RPC Service","0.01"},
      {rabbit_jsonrpc,"Rabbit JSON-RPC","2.3.1"},
      {rabbit_mochiweb,"RabbitMQ Mochiweb Embedding","2.3.1"},
      {mochiweb,"MochiMedia Web Server","1.3"},
      {crypto,"CRYPTO version 1","1.5.3"},
      {inets,"INETS  CXC 138 49","5.0.12"},
      {rabbit_stomp,"Embedded Rabbit Stomp Adapter","2.3.1"},
      {rabbit,"RabbitMQ","2.3.1"},
      {mnesia,"MNESIA  CXC 138 12","4.4.7"},
      {os_mon,"CPO  CXC 138 46","2.1.8"},
      {sasl,"SASL  CXC 138 11","2.1.5.4"},
      {stdlib,"ERTS  CXC 138 10","1.15.5"},
      {kernel,"ERTS  CXC 138 10","2.12.5"}]},
 {nodes,
     [{disc,['rabbit at site1']},
      {ram,['rabbit at site4','rabbit at site3','rabbit at site2']}]},
 {running_nodes,
     ['rabbit at site3','rabbit at site2','rabbit at site4',
      'rabbit at site1']}]
...done.

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.