[rabbitmq-discuss] RabbitMQ Instability in 3.0.1

[Dave Seltzer]

I have a RabbitMQ 3.0.1 cluster which I upgraded from 2.7.x

For the first couple of days it was quite stable.

This morning however I started noticing some odd behavior. Notably, I would
issue BasicPublish commands and they would run forever before finally
timing out. I attempted to restart the cluster and this ended with bad
results.

I finally got the cluster back online, but now its not behaving properly.

Node2 Starts up like this:

starting memory monitor
...done
-- core initialized
starting federation
...done
starting federation exchange decorator
 ...done
starting federation parameters
 ...done
starting federation upstream exchange type
 ...done
starting management agent
...done
starting HA policy validation
...done
starting policy parameters
 ...done
starting exchange, queue and binding recovery
...done
starting configured definitions
...done
starting empty DB check
...done
starting mirror queue slave sup                                       ...

BOOT FAILED
===========

Error description:
   {shutdown,
       {gen_server,call,
           [rabbit_sup,
            {start_child,
                {rabbit_mirror_queue_slave_sup,
                    {rabbit_mirror_queue_slave_sup,start_link,[]},
                    transient,infinity,supervisor,
                    [rabbit_mirror_queue_slave_sup]}},
            infinity]}}

Log files (may contain more information):
   /var/log/rabbitmq/rabbit at egressqueue02.log
   /var/log/rabbitmq/rabbit at egressqueue02-sasl.log

Stack trace:
   [{gen_server,call,3,[{file,"gen_server.erl"},{line,188}]},
    {rabbit_sup,start_supervisor_child,3,[]},
    {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},
    {rabbit,run_boot_step,1,[]},
    {rabbit,'-start/2-lc$^0/1-0-',1,[]},
    {rabbit,start,2,[]},
    {application_master,start_it_old,4,
                        [{file,"application_master.erl"},{line,274}]}]



BOOT FAILED
===========

Error description:
   {could_not_start,rabbit,
    {bad_return,
     {{rabbit,start,[normal,[]]},
      {'EXIT',
       {rabbit,failure_during_boot,
        {shutdown,
         {gen_server,call,
          [rabbit_sup,
           {start_child,
            {rabbit_mirror_queue_slave_sup,
             {rabbit_mirror_queue_slave_sup,start_link,[]},
             transient,infinity,supervisor,
             [rabbit_mirror_queue_slave_sup]}},
           infinity]}}}}}}}

Log files (may contain more information):
   /var/log/rabbitmq/rabbit at egressqueue02.log
   /var/log/rabbitmq/rabbit at egressqueue02-sasl.log

{"init terminating in
do_boot",{rabbit,failure_during_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot,{shutdown,{gen_server,call,[rabbit_sup,{start_child,{rabbit_mirror_queue_slave_sup,{rabbit_mirror_queue_slave_sup,start_link,[]},transient,infinity,supervisor,[rabbit_mirror_queue_slave_sup]}},infinity]}}}}}}}}}

I'm not sure how to proceed. Any advice would be hugely helpful.

Thanks!

-- 
Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130109/18ec3dd5/attachment.htm>