[rabbitmq-discuss] Scalability?

[Matthew Sackman]

Hi Wayne,

On Fri, May 07, 2010 at 07:48:54AM -0400, Wayne Van Den Handel wrote:
> I will test the basic consume, thanks for the tip. The machine I am running
> is a single socket 4 core with 4 gigs ram but there are other things running
> and taking up memory. Is there a gc interval that can be set to get memory
> back sooner? As I add messages and remove them I can see the exact memory
> taken with the admin tool and it keeps going up. It seems the rate at which
> I add to the queue exceeds the ability the clear memory. Basically even
> though there is little in the queue at any time the memory use keeps going
> up.

Rabbit assumes it gets the whole machine's memory to itself. If that's
not the case then you need to reduce the memory high watermark from 0.4
to some much lower figure - say, 0.1.
See http://www.rabbitmq.com/extensions.html#memsup

> At this point I am ready to give up until they harden the persister to
> go to disk as I assume the persister is the problem.

Yup. The new persister currently requires you build from source, but it
is actively being QA'd and great progress has been made getting it into
default. We've gone from about 100 hunks difference a couple of weeks
ago to now about 13 - so things are very much moving in this direction.

In the mean time, many people are already using the new persister, which
is on branch "bug21673" and if you're comfortable compiling from source
and using some code which has not yet completed our QA process, then we
would encourage you to use it, with the understanding that there may
still be bugs in it. Further battle testing is always welcome and should
you come across any bugs, any bug reports you make are very gratefully
received.

Matthew