[rabbitmq-discuss] Growing Memory Use and Queues

[Ilya Volodarsky]

Looking more through the queues, it looks like the queue memory sum adds up 
based on the most active queues. 

The more the queues grow in size, the more memory they take up - which 
makes sense. The problem is that erlang isn't properly reclaiming that 
memory once it is no longer used. I checked out the memory consumption for 
all the queues.

The top culprits (sorted by memory).

25152048
25151848
21038056
21038016
21038016
19373624
19373584
19373584

None of these queues have any queued messages at all really, though most of 
them have had queued messages at one time or another in the past. 

The funny thing is, after running the command a second time, many of those 
queues had dropped back to normal levels, or significantly decreased. In 
particular, one went from 24mb to 600K. It seems like some of them are at 
least *trying* to garbage collect.

I'm also seeing queues that have been heavily used in the past, but haven't 
had any data going through them for months. One is at 12.8 MB and another 
at 6.3 MB. This seems to indicate that garbage collection isnt happening 
for now-inactive queues over long periods of time.

It seems like a memory leak or bug in garbage collection. Looking at the 
changelog for the versions of RMQ I don't see anything super relevant that 
would fix by updating to 3.0.4.

Having fewer queues will probably help things, seems to me like a deeper 
problem with rabbitmq though?

On Tuesday, April 16, 2013 12:53:06 PM UTC-4, Ilya Volodarsky wrote:
>
> We're having some strange memory problems in our rabbit that looks like 
> it's attributed to our queues. We're running Rabbit 3.0.1 in production, 
> here's the management plugin overview: http://cl.ly/image/0z25241k320Q
>
> As you can see, we have almost no messages queued, but the server is using 
> 5.7GB of active memory. Here's the memory breakdown of the same node: 
> http://cl.ly/image/03381l2M3v2p
>
> The broker claims to be using 4.8 GB to store the 4454 queues. That's 
> strange because most of the queues claim to be using 14kb - 23kb, which 
> AFAIK, is the in the correct range for an idle queue. An average of 20kb 
> per queue * 4500 queues = 90 MB, not 4800 MB (53 times bigger). 
>
> Here's the JSON description of a queue from the /queues call:
>
> {"memory":14624,"idle_since":"2013-04-15 
> 22:45:20","policy":"","exclusive_consumer_tag":"","messages_ready":0,"messages_unacknowledged":0,"messages":0,"consumers":4,"active_consumers":4,"backing_queue_status":{"q1":0,"q2":0,"delta":["delta","undefined",0,"undefined"],"q3":0,"q4":0,"len":0,"pending_acks":0,"target_ram_count":0,"ram_msg_count":0,"ram_ack_count":0,"next_seq_id":0,"persistent_count":0,"avg_ingress_rate":0.0,"avg_egress_rate":0.0,"avg_ack_ingress_rate":0.0,"avg_ack_egress_rate":0.0},"messages_details":{"rate":0,"interval":90165823981,"last_event":1366065920939},"messages_ready_details":{"rate":0,"interval":90165823981,"last_event":1366065920939},"messages_unacknowledged_details":{"rate":0,"interval":90165823981,"last_event":1366065920939},"name":"Ingestion.1899vw88el","vhost":"/segment","durable":true,"auto_delete":false,"arguments":{},"node":"rabbit at rabbit2"}
>
> A few things to note: we have 4 servers subscribing to every queue, and 
> because of the RMQ client (node-amqp) we're using, it creates a channel per 
> subscription. I realize that's not ideal and we'll be fixing the client to 
> make multiple subscriptions per channel, but either way, the broker reports 
> only 350 MB of the memory use coming from the connections vs. 4.8 GB for 
> queues. 
>
> The broker has also been up for 76 days without downtime, so maybe there's 
> some kind of memory leak? We're seeing the memory go up over time. 
>
> Thanks!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130417/40ef0c05/attachment-0001.htm>