[rabbitmq-discuss] Scalability?

Fri May 7 14:23:01 BST 2010

Wayne Van Den Handel <wvandenhandel at dataraker.com> writes:
>    Mathew explained everything with the high memory watermark setting. The
>    machine is not dedicated to RabbitMQ and is running other things.
>    Erlang thinks it can have 40% of memory which I guess can go to 80%
>    before forcing GC. On this machine RabbitMQ crashes well before it
>    reaches 80% as that much memory is not available.
>    I argue that is a poor assumption for a default setting. Nothing should
>    assume out of the box it can have 80% of physical memory (regardless of
>    what is running). I hope the new persister takes away some of those
>    assumptions. It should be smart enough to work with the memory
>    available.

Matthew knows more about Erlang behaviour than I do, but I don't find
that explanation very satisfying, given your description of the
problem.  It sounds odd that the Erlang VM would expand to the memory
limit before attempting a GC.  I would expect it to GC long before
that limit was reached, so that in your case, the limit never would be
reached.

And although the default memory limit percentage is based on the
assumption that rabbit has the machine to itself, it's certainly not
my experience that you need to dedicate a machine to rabbit for it to
operate robustly.

So if you are willing to invest the time in turning your python code
into something you are willing to share with us, I'd be happy to look
at it.

David

-- 
David Wragg
Staff Engineer, RabbitMQ
SpringSource, a division of VMware