[rabbitmq-discuss] Memory grows very fast

Thu May 23 19:58:52 BST 2013

Artur,

On 23/05/13 15:44, rabbit15 wrote:
> I see that on faster machine which has 32 CPU(Intel(R) Xeon(R) CPU E5-2670 0
> @ 2.60GHz) and 64GB RAM there can be store more messages about 10 milions.
> Previous machine was with 24 CPU (Intel(R) Xeon(R) CPU           X5675  @
> 3.07GHz) and 92 GB RAM and there could 600k -2500k stored messages.

That's probably just due to the changes in timing.

You are clearly putting rabbit under extreme load. In the status report 
you just posted, the system had been running for just ~12 minutes, yet 
there were 800+ open connections, taking up 1.3GB of memory between 
them. Now, connections only hold data very briefly, so this is a clear 
indication that the rabbit is heavily loaded. Furthermore, in an earlier 
report you posted, the run_queue sat at 148 - that's 148 Erlang 
processes waiting for a CPU to become free to do some work.

With such a usage, memory consumption will always be high.

> Maybe there is some option for GC ?

There are plenty of Erlang gc options to tweak. Usually that makes 
matters worse, but if you really want to go down that route then, for 
starters, try setting the ERL_FULLSWEEP_AFTER environment variable to 0 
or some other low value (the default is 65535) in 
/etc/rabbitmq/rabbitmq-env.conf, i.e.

export ERL_FULLSWEEP_AFTER 0

and restart rabbit.

You can confirm the setting took effect by running

$ rabbitmqctl eval 'erlang:system_info(fullsweep_after).'
{fullsweep_after,0}
...done.
$

Regards,

Matthias.