[rabbitmq-discuss] Erlang crashes trying to allocate 583848200 bytes of memory

[Jerry Kuch]

Hi, Dom...

> we have a similar problem that can be summarised as follows.  Our goal
> is to compare RabbitMQ (the last stable version) to ActiveMQ.  To this
> end we set up two minimal and similar testing environments having both
> 1 publisher, 1 subscriber and 1 message broker (RabbitMQ/ ActiveMQ)
> server.  In both testing environments the message broker runs under a
> Windows Hyper-V virtual machine (hosted on a Windows Server 23008 64
> bits) that has as OS a 32 bit Windows Server 2008 (if I'm correct we
> allocated 2 GB, perhpas 4 GB, to the VM and 1 CPU).  The
> publishers/subscribers are run under Windows XP/Vista and are on the
> same LAN as the broker VM.

This ought to be an entirely reasonable set up.  Apologies that you're
having trouble with it.  There's definitely something awry here and
current evidence points toward it not being yiour fault.

> On the subscriber side it connects to the broker waiting for messages.
> When a new chuck started message is received, it saves the received
> timestamp and then counts how many messages it has received till the
> indication of the chunk end message. At this point it reads the system
> clock and calculates how much time has elapsed since the chunck start
> on the publisher side (as reported in the received timestamp) and the
> arrival on the chunk end message on the subscriber side.  The
> subscriber then logs all these data (start timestamp, end timestamp,
> elapsed time, number of messages), and is ready to process a new
> message chunk.  Anyway further details and the Java code is available
> in the indicated RabbitMQ newsgroup's message.  In our testing
> environment, therefore, there are no durable queues, no permanent
> messages, a single subscriber, ...

Our of curiosity when and how is your consumer ACK-ing the messages it
gets?  Do you have auto-ACK set?  Or are you explicitly ACK-ing them,
either individually or as a range?

> What we experienced are crashes of the RabbitMQ server due to the
> memory allocation issues. Indeed, the Windows Task Manager shows the
> RabbitMQ process increasing memory, until it crashes.  On the RabbitMQ
> forum someone suggested to my colleague Cristoforo to reduce the
> vm_memory_high_watermark value from its default 0.40 value.  We tried
> with 0.25, 0.20 and 0.10. The crash problem doesn't disappear unless
> the subscriber runs on a fast machine (e.g. putting the subscriber on
> the same VM as the message broker seems to have less crashing issues).
> With ActiveMQ we don't experience similar issues at all.  We are not
> using Erlnag under a 64 bit OS (our VM, as stated, is a 32 bit Windows
> Server 2008), and the vm_memory_high_watermark has been lowered enough
> (we think).

So your observations are that when you have a fast consumer, your
broker fares better?

> Our conclusion is that the Windows Erlang implementation has some
> serious issues on memory management under Windows, or that we have not
> configured Erlang/RabbitMQ in a proper way.

At the moment, it looks like the former is very likely true.  Some on
the Rabbit team have had some interaction with Erlang's maintainers on
the issue, but we don't have a good answer for you yet.

> Sorry for the long comment, but I think it was necessary to describe a
> testing scenario that, in my opinion, can shed some lights on the
> flagged issue.

No apologies necessary...  your feedback is helpful, and has helped us
confirm that something is awry on Windows.  The Rabbit user base seems
rather tilted toward Unix so when there are problems specific to
running on Windows, we don't always find out as quickly as we might
like...

Best regards,
Jerry