[rabbitmq-discuss] clearing a queue error

Sat Sep 28 21:46:09 BST 2013

There should be 4 durable queues on 4 different vhosts, so I'm not sure what happened. The durable queues are typically empty - they only start to fill up if the consumer is down for whatever reason.

I think I mucked around enough that I can't really give you a good error report to track down, unfortunately. So, I broke down the cluster, cleared out all the mnesia data from server 2, deleted the mnesia queue files/directories from server 1, and brought server 1 back up. Server 1 came up ok and I re-created the durable queues there and it seems to be running and routing messages ok. I am seeing several instances of one error in the log file:

=ERROR REPORT==== 28-Sep-2013::10:56:31 ===
** Generic server rabbit_mgmt_external_stats terminating 
** Last message in was emit_update
** When Server state == {state,1024}
** Reason for termination == 
** {{badmatch,[]},
    [{rabbit_mgmt_external_stats,i,2,
         [{file,
              "rabbitmq-management-agent/src/rabbit_mgmt_external_stats.erl"},
          {line,146}]},
     {rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,
         [{file,
              "rabbitmq-management-agent/src/rabbit_mgmt_external_stats.erl"},
          {line,143}]},
     {rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,
         [{file,
              "rabbitmq-management-agent/src/rabbit_mgmt_external_stats.erl"},
          {line,143}]},
     {rabbit_mgmt_external_stats,emit_update,1,
         [{file,
              "rabbitmq-management-agent/src/rabbit_mgmt_external_stats.erl"},
          {line,275}]},
     {rabbit_mgmt_external_stats,handle_info,2,
         [{file,
              "rabbitmq-management-agent/src/rabbit_mgmt_external_stats.erl"},
          {line,263}]},
     {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

And the management interface isn't behaving well - the Connections, Channels, Exchanges, Queues pages give me the error below:

{error,
    {error,
        {badrecord,dict},
        [{dict,get_slot,2,[{file,"dict.erl"},{line,396}]},
         {dict,find,2,[{file,"dict.erl"},{line,139}]},
         {rabbit_mgmt_util,id0,2,
             [{file,"rabbitmq-management/src/rabbit_mgmt_util.erl"},
              {line,260}]},
         {rabbit_mgmt_util,vhost,1,
             [{file,"rabbitmq-management/src/rabbit_mgmt_util.erl"},
              {line,116}]},
         {rabbit_mgmt_util,filter_conn_ch_list,3,
             [{file,"rabbitmq-management/src/rabbit_mgmt_util.erl"},
              {line,443}]},
         {rabbit_mgmt_wm_connections,to_json,2,
             [{file,"rabbitmq-management/src/rabbit_mgmt_wm_connections.erl"},
              {line,36}]},
         {webmachine_resource,resource_call,3,[]},
         {webmachine_resource,do,3,[]}]}}

Any suggestions on how to fix this one? I did try to delete the -plugins-expand mnesia directory and restart in case there was some bad data in there, but that didn't do anything.

Thanks again,

Warren

> -----Original Message-----
> From: Matthias Radestock [mailto:matthias at rabbitmq.com]
> Sent: Saturday, September 28, 2013 12:57 AM
> To: Warren Smith
> Cc: Discussions about RabbitMQ
> Subject: Re: [rabbitmq-discuss] clearing a queue error
> 
> Warren,
> 
> On 28/09/13 00:58, Warren Smith wrote:
> > I'll send you the mnesia directory.
> 
> Are you sure that's from the node on which you see the error?
> 
> There are just two durable queues and they are completely empty. I
> can't see how the error would arise under those circumstances.
> 
> Regards,
> 
> Matthias.