[rabbitmq-discuss] Disk/Memory Usage with RabbitMQ

[Matthias Radestock]

Barry,

Barry Pederson wrote:

> After restarting the server just now, memory usage was back down, and 
> the rabbit_persister.LOG file was fairly small (62k), where the 
> rabbit_persister.LOG.previous is about 2.1mb

Good. That's what I expected see.

> I'm fairly certain though that on another occasion a week or so ago, a 
> restarted server came back up with large memory usage and a big 
> rabbit_persister.LOG - I ended up wiping out the mnesia directory to get 
> things back to square one.

When RabbitMQ restarts it goes through the persister log and figures out 
which messages need (re)delivering. When the persister log is large this 
can take some time and can also eat up a lot of memory. Once that's done 
though, and assuming there aren't lots of undelivered messages, the 
memory usage should drop significantly and a new, shorter persister log 
should be written.

>> Try
>>
>> [process_info(P) || P <- processes(),
>>                     process_info(P, memory) > {memory, 100000}].
>>
>> which will list the details of all processes with a memory consumption 
>> greater than 100000 bytes.
> [...]
> Looks like "heap_size" and "reductions" are growing.

...of the rabbit_gc_persist process, which keeps track of undelivered 
persistent messages. I get the same results on my system, so I I am 
pretty confident that I have reproduced the problem you are seeing.

>>> Would publishing persistent messages to a durable exchange with no 
>>> queues bound to it at all (yet) cause the messages to stick around?
>>
>> You might be on to something there. My own testing shows a surprising 
>> growth in memory and the persister log in this scenario. I will do 
>> some more digging.
> 
> Ah good, I'm glad it's not just me.

The problem turns out to be that undeliverable persistent messages stick 
around in the rabbit_gc_persist process memory and the persister log 
until the node restart.

The fix for this will be in the next RabbitMQ release. If you don't want 
to wait that long, and are feeling adventurous, you can try the most 
recent snapshot from http://dev.rabbitmq.com/snapshots/rabbitmq/


Thanks for reporting and helping me track down this problem.


Matthias.