[rabbitmq-discuss] Memory watermark alert not resetting.

Thu Oct 4 16:57:35 BST 2012

So that looks to me like mongo-qa2 is a RAM node, and you are starting 
it before mongo-qa1, which is presumably a disc node?

What happens if you start mongo-qa1 first?

Cheers, Simon

On 04/10/12 16:38, Raviv Pavel wrote:
> Upgraded the other node (there are two) and it fails to start.
> Here is the startup log:
>
>
> +---+   +---+
> |   |   |   |
> |   |   |   |
> |   |   |   |
> |   +---+   +-------+
> |                   |
> | RabbitMQ  +---+   |
> |           |   |   |
> |   v2.8.7  +---+   |
> |                   |
> +-------------------+
> AMQP 0-9-1 / 0-9 / 0-8
> Copyright (C) 2007-2012 VMware, Inc.
> Licensed under the MPL.  See http://www.rabbitmq.com/
>
> node           : rabbit at mongo-qa2
> app descriptor :
> /usr/lib/rabbitmq/lib/rabbitmq_server-2.8.7/sbin/../ebin/rabbit.app
> home dir       : /var/lib/rabbitmq
> config file(s) : (none)
> cookie hash    : pu0BlS9+G2N9yLfd51TkmA==
> log            : /var/log/rabbitmq/rabbit at mongo-qa2.log
> sasl log       : /var/log/rabbitmq/rabbit at mongo-qa2-sasl.log
> database dir   : /var/lib/rabbitmq/mnesia/rabbit at mongo-qa2
> erlang version : 5.8.4
>
> -- rabbit boot start
> starting file handle cache server
> ...done
> starting worker pool
>   ...done
> starting database                                                     ...
>
> BOOT FAILED
> ===========
>
> Error description:
>     {error,{failed_to_cluster_with,['rabbit at mongo-qa1'],
> "Mnesia could not connect to any disc nodes."}}
>
> Log files (may contain more information):
>     /var/log/rabbitmq/rabbit at mongo-qa2.log
>     /var/log/rabbitmq/rabbit at mongo-qa2-sasl.log
>
> Stack trace:
>     [{rabbit_mnesia,init_db,3},
>      {rabbit_mnesia,init,0},
>      {rabbit,'-run_boot_step/1-lc$^1/1-1-',1},
>      {rabbit,run_boot_step,1},
>      {rabbit,'-start/2-lc$^0/1-0-',1},
>      {rabbit,start,2},
>      {application_master,start_it_old,4}]
>
> {"Kernel pid
> terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot}}}}}"}
>
>
> Thanks
> /--/Raviv
> /
> /
>
>
>
> On Thu, Oct 4, 2012 at 5:34 PM, Simon MacMullen <simon at rabbitmq.com
> <mailto:simon at rabbitmq.com>> wrote:
>
>     Hard to believe that's to do with the server not starting - can you
>     post a more complete log?
>
>
>     On 04/10/12 16:32, Raviv Pavel wrote:
>
>         After upgrading the server won't start. from the logs:
>
>         =ERROR REPORT==== 4-Oct-2012 <tel:2012>::17:11:59 ===
>         ** Generic server <0.341.0 <tel:0.341.0>> terminating
>         ** Last message in was {'$gen_cast',
>                                      {method,
>
>            {'queue.declare',0,<<"es1">>,__false,true,false,
>                                              false,false,
>
>            [{<<"x-ha-policy">>,longstr,<<__"all">>}]},
>                                          none,noflow}}
>
>
>         Thanks
>         /--/Raviv
>         /
>
>         /
>
>
>
>         On Thu, Oct 4, 2012 <tel:2012> at 5:05 PM, Raviv Pavel
>         <raviv at gigya-inc.com <mailto:raviv at gigya-inc.com>
>         <mailto:raviv at gigya-inc.com <mailto:raviv at gigya-inc.com>>> wrote:
>
>              All queues are empty and "rabbitmqctl list_queues name
>         memory" shows
>              they use about 3K each.
>              VM memory stats are much lower then the one shown in the
>         overview.
>              We're using 2.8.1 - I'll try upgrading to 2.8.7
>
>              Thanks
>              /--/Raviv
>              /
>              /
>
>
>
>              On Thu, Oct 4, 2012 <tel:2012> <tel:2012 <tel:2012>> at
>         4:57 PM, Matthias Radestock
>
>         <matthias at rabbitmq.com <mailto:matthias at rabbitmq.com>
>         <mailto:matthias at rabbitmq.com <mailto:matthias at rabbitmq.com>>__>
>         wrote:
>
>                  On 04/10/12 15:53, Simon MacMullen wrote:
>
>                      On 04/10/12 13:01, Raviv Pavel wrote:
>
>                          Based on the management UI, memory usage
>         doesn't drop
>                          and publishers are
>                          still blocked.
>
>
>                      Hmm. In that case my first guess would be that the
>         queue you
>                      deleted /
>                      purged wasn't the queue which was using all the memory.
>                      Check the memory
>                      use of other queues - this can be found on the
>         queue details
>                      page, or
>                      with "rabbitmqctl list_queues name memory". Also,
>         check the
>                      VM-wide
>                      memory statistics (on the node details page, under
>         "advanced". This
>                      might give a clue as to where the memory is going.
>
>
>                  ...and please make sure you are running the latest
>         version of
>                  RabbitMQ (2.8.7, atm), since we have fixed a few memory
>         leaks in
>                  the in recent versions.
>
>                  Matthias.
>
>
>
>
>
>     --
>     Simon MacMullen
>     RabbitMQ, VMware
>
>

-- 
Simon MacMullen
RabbitMQ, VMware