[rabbitmq-discuss] Mgmt API stopped reporting a lot of stuff

Fri Apr 19 00:37:21 BST 2013

So tiger03 was the node that had the stats db, which crashed, it couldn't be
restarted gracefully either, so did a "kill -9". The stats db moved to
tiger01, so everything works fine there now, but tiger02 reports: 

=ERROR REPORT==== 18-Apr-2013::23:14:56 ===
webmachine error: path="/api/overview"
{error,{error,{badmatch,false},
              [{rabbit_mgmt_wm_overview,version,1},
               {rabbit_mgmt_wm_overview,to_json,2},
               {webmachine_resource,resource_call,3},
               {webmachine_resource,do,3},
               {webmachine_decision_core,resource_call,1},
               {webmachine_decision_core,decision,1},
               {webmachine_decision_core,handle_request,2},
               {rabbit_webmachine,'-makeloop/1-fun-0-',2}]}}

When tiger03 started again the log contains a lot of these:

=ERROR REPORT==== 18-Apr-2013::23:14:36 ===
** Generic server <0.6527.4> terminating
** Last message in was {init,<0.3561.0>}
** When Server state == {q,{amqqueue,
                               {resource,<<"vhost1">>,queue,
                                   <<"tmp_topic-0.1643550938460976">>},
                               true,true,<0.12110.2>,[],<0.6527.4>,[],[],
                               [{vhost,<<"vhost1">>},
                                {name,<<"HA">>},
                                {pattern,<<".*">>},
                                {definition,[{<<"ha-mode">>,<<"all">>}]},
                                {priority,0}],
                               [{<4861.12913.2>,<4861.12912.2>},
                                {<4637.186.1>,<4637.185.1>},
                                {<0.15006.2>,<0.12987.2>}]},
                           none,false,undefined,undefined,
                           {[],[]},
                           undefined,undefined,undefined,undefined,
                           {state,fine,5000,undefined},
                           {0,nil},
                           undefined,undefined,undefined,
                           {dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []}}},
                           1,
                           {{0,nil},{0,nil}},
                           undefined,
                           {dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []}}},
                           undefined,undefined}
** Reason for termination == 
** {'module could not be loaded',
       [{undefined,init,
            [{amqqueue,
                 {resource,<<"vhost1">>,queue,
                     <<"tmp_topic-0.1643550938460976">>},
                 true,true,<0.12110.2>,[],<0.6527.4>,[],[],
                 [{vhost,<<"vhost1">>},
                  {name,<<"HA">>},
                  {pattern,<<".*">>},
                  {definition,[{<<"ha-mode">>,<<"all">>}]},
                  {priority,0}],
                 [{<4861.12913.2>,<4861.12912.2>},
                  {<4637.186.1>,<4637.185.1>},
                  {<0.15006.2>,<0.12987.2>}]},
             true,#Fun<rabbit_amqqueue_process.5.64830354>]},
        {rabbit_amqqueue_process,handle_call,3},
        {gen_server2,handle_msg,2},
        {proc_lib,wake_up,3}]}

and it stopped at "starting adding mirrors to queues                                    
...", but most notably, the whole cluster stopped deliver any messages!

The sasl log for tiger03 also contains:

=SUPERVISOR REPORT==== 18-Apr-2013::23:14:36 ===
     Supervisor: {local,rabbit_amqqueue_sup}
     Context:    child_terminated
     Reason:     {undef,
                     [{undefined,init,
                          [{amqqueue,
                               {resource,<<"vhost1">>,queue,
                                   <<"tmp_topic-0.1643550938460976">>},
                               true,true,<0.12110.2>,[],<0.6527.4>,[],[],
                               [{vhost,<<"vhost1">>},
                                {name,<<"HA">>},
                                {pattern,<<".*">>},
                                {definition,[{<<"ha-mode">>,<<"all">>}]},
                                {priority,0}],
                               [{<4861.12913.2>,<4861.12912.2>},
                                {<4637.186.1>,<4637.185.1>},
                                {<0.15006.2>,<0.12987.2>}]},
                           true,#Fun<rabbit_amqqueue_process.5.64830354>]},
                      {rabbit_amqqueue_process,handle_call,3},
                      {gen_server2,handle_msg,2},
                      {proc_lib,wake_up,3}]}
     Offender:   [{pid,<0.6527.4>},
                  {name,rabbit_amqqueue},
                  {mfa,
                      {rabbit_amqqueue_process,start_link,
                          [{amqqueue,
                               {resource,<<"vhost1">>,queue,
                                   <<"tmp_topic-0.1643550938460976">>},
                               true,true,<0.12110.2>,[],<0.12987.2>,[],[],
                               [{vhost,<<"vhost1">>},
                                {name,<<"HA">>},
                                {pattern,<<".*">>},
                                {definition,[{<<"ha-mode">>,<<"all">>}]},
                                {priority,0}],
                               [{<4861.12913.2>,<4861.12912.2>},
                                {<4637.186.1>,<4637.185.1>},
                                {<0.15006.2>,<0.12987.2>}]}]}},
                  {restart_type,temporary},
                  {shutdown,4294967295},
                  {child_type,worker}]

--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Mgmt-API-stopped-reporting-a-lot-of-stuff-tp26065p26092.html
Sent from the RabbitMQ mailing list archive at Nabble.com.