[rabbitmq-discuss] Excessive RabbitMQ memory consumption

[Matt Pietrek]

We have a RabbitMQ cluster in production that seems to be using an
excessive amount of memory. I'm trying to figure out what sort of
diagnostics are available to us.

Background: RabbitMQ 3.0.2, Erlang R15B01 on Ubuntu 10.04 (64-bit).

Yes, we'd love to upgrade to 3.2.1 and the latest Erlang, but this is in
production, and we can't just throw something untested (by us, in our
scenario) into production. So we need to understand what's wrong now, and
then we can test on newer releases to validate things are better.

We've now upped our VM to 8GB, and RabbitMQ/Erlang seem more than happy to
just eat up more memory and keep going. Earlier today a node crashed when
the VM had 4GB, and Rabbit tried to allocate 2.2GB (this was the last entry
in our logs). Nothing else of consequence is running on this VM.

Currently the story looks like this:

Roughly 300 queues. Traffic is about 50 messages/second, although at peak
times it goes to 400/second. However, message rates and memory consumption
seem unrelated. Messages are at most 1KB in size. Connections are steady at
600, with 800 channels. We use message acks. Most messages are unacked for
a few milliseconds before they're acked.

Currently Rabbit is consuming in the neighborhood of 1.8GB (via the web ui)
on average. Here's a breakdown of the node reported memory usage details
from the UI:

Connections 154.9MB
Queues 16.2MB
Plugins 89.6kB
Other process memory 14.2MB
Mnesia 2.6MB
Message store index 2.2MB
Management database  1.3GB
Other ETS tables 6.9MB
Binaries 115.9MB
Code 17.5MB
Atoms 654.9kB
Other system 36.1MB

The management database at 1.3GB seems excessive.

Watching the memory consumption over time, it might drop down to 1 GB, then
peak at 2.4GB. It seems to randomly wander between those values over time.
We have our watermark set at 60%, so 4.7GB there.

Needless to say, this is baffling and concerning.

Are their known issues with this configuration? How can we pin down what's
happening?

Thanks,

Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131120/d6b5caf1/attachment.htm>