[rabbitmq-discuss] Unable to start RabbitMQ server after crash because of vm_memory_high_watermark alarm.

[ChrisStoyles]

Hi Matthew,

Yes we were running 1.7.2 :) we are planning to upgrade just as soon
as things settle down after this event.

Thanks for posting, I actually didn't get an email alert from Google
Groups that a new message had been posted hence the delay in getting
back to you.

It turns out our problem was to do with a bug in Erlang which was
noted and patched here:
http://erlang.2086793.n4.nabble.com/Patch-for-18-exabyte-memory-allocation-failure-td3342133.html

We downloaded the latest stable release of Erlang and compiled from
source. After this RabbitMQ came up without any problems whatsoever,
turns out this was an Erlang bug not RabbitMQ (because Ubuntu 10.04LTS
packages a very old version of both RabbitMQ and Erlang).

Thanks for your help.

Chris.

On Aug 23, 7:16 pm, Matthew Sackman <matt... at rabbitmq.com> wrote:
> Hi Chris,
>
> On Sun, Aug 21, 2011 at 05:08:39PM -0700, ChrisStoyles wrote:
> > I need some advice on moving /var/lib/rabbitmq to a new server with
> > more RAM. What has happened is that on our 8GB RAM server we've had a
> > surge in activity and one of our queues has grown much faster than it
> > can be consumed by our "workers".
>
> > This has resulted in RabbitMQ crashing, with an erl_crash.dump file
> > appearing.
>
> > At present when I attempt to start the server up again, it tries to
> > roll the persister.LOG file and recover what was in the queue at the
> > time of the crash. However even with rabbitmq.config set up as below,
> > the vm_memory_high_watermark alarm is triggered and erl_crash.dump is
> > created all over again.
>
> Erm, persister.LOG suggests a RabbitMQ pre 2.0.0. Is that true? We would
> very strongly recommend you upgrade asap.
>
> > [
> >   {rabbit, [{vm_memory_high_watermark, 0.9}]}
> > ].
>
> Yeah, that's not going to make any difference to pre 2.0 Rabbits. Pre
> 2.0, all msgs are always held in RAM.
>
> > I've got another server with 24GB of RAM, so I set it's HOSTNAME to
> > the same as the 8GB server and installed the same version of RabbitMQ.
> > I then copied over the contents of /var/lib/rabbitmq and attempted to
> > start RabbitMQ. I no longer hit the vm_memory_high_watermark however
> > Erlang seems to close/shutdown while attempting to recover
> > persister.LOG.
>
> That's odd - if the hostname and the nodename of Rabbit is the same, it
> should start up I think. Is there anything in the logs?
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss