[rabbitmq-discuss] Rabbitmq 2.7.1 / R14B02 crash

[Andrei Vereha]

On Wed, 21 Mar 2012 17:22:07 +0000
Simon MacMullen <simon at rabbitmq.com> wrote:

> The erl_crash.dump is not terribly useful I'm afraid.
> 
> How much memory did the machine have?
> 
> What was the high watermark set to?

Quoting logs:
Memory limit set to 1585MB of 3963MB total.

> Can we see the logs?
> 

The logs are here:
http://dl.transfer.ro/logs-transfer_RO-22mar-8ea6e9.7z

The server started at :
=INFO REPORT==== 20-Mar-2012::19:15:20 ===
and it crashed at:
Wed Mar 21 13:05:19 2012
The startup_log is from the last startup(after 13:05).
The last entry in the -sasl file is from 28-Feb-2012.
The previous runs that you see in logs was a 2.8.0 install, but it
seemed to leak memory. That's why I downgraded it to 2.7.1.

Regarding 2.7.1, it crashed a few times before but killed by OOM killer.
This was the output in dmesg:

[4428010.034014] Out of memory: Kill process 18725 (beam.smp) score 855
or sacrifice child
[4428010.034117] Killed process 18725 (beam.smp) total-vm:4978220kB,
anon-rss:3319964kB, file-rss:0kB
[4428010.046305] beam.smp invoked oom-killer: gfp_mask=0x200da,
order=0, oom_adj=0, oom_score_adj=0
[4428010.046312] beam.smp cpuset=/ mems_allowed=0
[4428010.046315] Pid: 18777, comm: beam.smp Not tainted 3.0.0-12-server
#20-Ubuntu

Here is the "normal" memory usage(reported by rabbtimqctl status):

 {memory,
     [{total,283178496},
      {processes,153323688},
      {processes_used,151332168},
      {system,129854808},
      {atom,1575153},
      {atom_used,1552033},
      {binary,14064320},
      {code,14557226},
      {ets,22433696}]},
 {vm_memory_high_watermark,0.39999999985562695},
 {vm_memory_limit,1662359961}]

Thank you,
 Andrei

> Cheers, Simon
> 
> On 21/03/12 17:14, Andrei Vereha wrote:
> > Hello ,
> >
> > A few hours ago our Rabbitmq server crashed with message:
> >
> > =erl_crash_dump:0.1
> > Wed Mar 21 13:05:19 2012
> > Slogan: eheap_alloc: Cannot allocate 1824525600 bytes of memory (of
> > type "heap").
> > System version: Erlang R14B02 (erts-5.8.3) [source]
> > [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]
> > Compiled: Fri May  6 16:15:00 2011 Taints:
> >
> > I uploaded the compressed erl_crash.dump file here(uncompressed it
> > takes 1.5GB):
> >
> > http://dl.transfer.ro/dump-transfer_RO-21mar-9ef325.7z
> >
> > When this happened vm_memory_high_watermark was not 'activated':
> > grep vm_memory_high_watermark rabbit at rabbitmq-01.log ->  prints
> > nothing.
> >
> > Why did it crashed?
> >
> > Thank you,
> >    Andrei
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
>