<div dir="ltr">We have a RabbitMQ cluster in production that seems to be using an excessive amount of memory. I'm trying to figure out what sort of diagnostics are available to us.<div><br></div><div>Background: RabbitMQ 3.0.2, Erlang R15B01 on Ubuntu 10.04 (64-bit).</div>
<div><br></div><div>Yes, we'd love to upgrade to 3.2.1 and the latest Erlang, but this is in production, and we can't just throw something untested (by us, in our scenario) into production. So we need to understand what's wrong now, and then we can test on newer releases to validate things are better.</div>
<div><br></div><div>We've now upped our VM to 8GB, and RabbitMQ/Erlang seem more than happy to just eat up more memory and keep going. Earlier today a node crashed when the VM had 4GB, and Rabbit tried to allocate 2.2GB (this was the last entry in our logs). Nothing else of consequence is running on this VM.</div>
<div><br></div><div>Currently the story looks like this:</div><div><br></div><div>Roughly 300 queues. Traffic is about 50 messages/second, although at peak times it goes to 400/second. However, message rates and memory consumption seem unrelated. Messages are at most 1KB in size. Connections are steady at 600, with 800 channels. We use message acks. Most messages are unacked for a few milliseconds before they're acked.</div>
<div><br></div><div>Currently Rabbit is consuming in the neighborhood of 1.8GB (via the web ui) on average. Here's a breakdown of the node reported memory usage details from the UI:</div><div><br></div><div><div>Connections <span class="" style="white-space:pre"> </span>154.9MB</div>
<div>Queues <span class="" style="white-space:pre"> </span>16.2MB</div><div>Plugins <span class="" style="white-space:pre"> </span>89.6kB</div><div>Other process memory <span class="" style="white-space:pre"> </span>14.2MB</div>
<div>Mnesia <span class="" style="white-space:pre"> </span>2.6MB</div><div>Message store index <span class="" style="white-space:pre"> </span>2.2MB</div><div>Management database <span style="white-space:pre"> </span>1.3GB</div>
<div>Other ETS tables <span class="" style="white-space:pre"> </span>6.9MB</div><div>Binaries <span class="" style="white-space:pre"> </span>115.9MB</div><div>Code <span class="" style="white-space:pre"> </span>17.5MB</div>
<div>Atoms <span class="" style="white-space:pre"> </span>654.9kB</div><div>Other system <span class="" style="white-space:pre"> </span>36.1MB</div></div><div><br></div><div>The management database at 1.3GB seems excessive.</div>
<div><br></div><div>Watching the memory consumption over time, it might drop down to 1 GB, then peak at 2.4GB. It seems to randomly wander between those values over time. We have our watermark set at 60%, so 4.7GB there.</div>
<div><br></div><div>Needless to say, this is baffling and concerning.</div><div><br></div><div>Are their known issues with this configuration? How can we pin down what's happening?</div><div><br></div><div>Thanks,</div>
<div><br></div><div>Matt</div></div>