[rabbitmq-discuss] beam instances and growing memory usage

[Matthias Radestock]

Matt,

matt wrote:
> When we did a complete site restart on Tuesday night, Wednesday worked 
> flawlessly.  We rebuilt the cluster (disk based nodes) and created a 
> total of 7 brokers on the box.

When you are saying "disk based nodes", do you mean that all broker 
nodes are configured as disk nodes? That is not good for performance. If 
you have all nodes on a single machine and they all write to the same 
disk then there is no point having more than one disk node.

> When I was watching the instances on Wednesday, each instance was 
> taking up about 0.1% of the server ram, according to top. I checked
> yesterday (Thursday) and each instance was taking 2.0%. I just looked
> now (Friday) and it's taking 2.9%. Durable is set to false - so it
> doesn't appear that things should be persisting anywhere. Any ideas
> why it'd be growing over time?

The growth is not necessarily anything to worry about. You still have 
plenty of memory left and the growth might stop after a while. Keep an 
eye on it.

It is worth checking the queue sizes. Connect to one of the nodes in 
your cluster - any node will do - and run rabbit_amqqueue:stat_all(), e.g.

$ erl -sname debug -hidden -remsh rabbit at tanto
Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] [hipe] 
[kernel-poll:false]
Eshell V5.5.5  (abort with ^G)
(rabbit at tanto)1> rabbit_amqqueue:stat_all().
[{ok,{resource,<<"/">>,queue,<<"test queue">>},0,0}]
(rabbit at tanto)2> CTRL-c CTRL-c

That will list all the queues in the system, their message queue length 
and their number of consumers.


On another topic, from your 'top' reports I notice that although the CPU 
load is unevenly distributed over the nodes, the distribution changes 
over time and the cumulative CPU times are much more even. So it looks 
like your load-balancing logic is working fine over the long run. 
Curiously though there are two nodes in your latter report which have 
accumulated much less CPU time.


Matthias.