[rabbitmq-discuss] Growing Memory Use and Queues

[harel at thisisglow.com]

We're seeing this recently since upgrading to the 3.x version of rabbit. 
We are running about 25 queues and all is good for a while. But sometimes 
rabbit crashes hard with out of memory errors even if the queues are not 
noticeably larger or busier than usual. I have a 3.2GB crash dump which 
honestly I don't even know how to begin looking at. 
This is running on a 8 core 16GB  server. 
How do I approach this? is it a bug in rabbit or something I can do ?
Thanks ,
Harel

On Tuesday, April 16, 2013 5:53:06 PM UTC+1, Ilya Volodarsky wrote:
>
> We're having some strange memory problems in our rabbit that looks like 
> it's attributed to our queues. We're running Rabbit 3.0.1 in production, 
> here's the management plugin overview: http://cl.ly/image/0z25241k320Q
>
> As you can see, we have almost no messages queued, but the server is using 
> 5.7GB of active memory. Here's the memory breakdown of the same node: 
> http://cl.ly/image/03381l2M3v2p
>
> The broker claims to be using 4.8 GB to store the 4454 queues. That's 
> strange because most of the queues claim to be using 14kb - 23kb, which 
> AFAIK, is the in the correct range for an idle queue. An average of 20kb 
> per queue * 4500 queues = 90 MB, not 4800 MB (53 times bigger). 
>
> Here's the JSON description of a queue from the /queues call:
>
> {"memory":14624,"idle_since":"2013-04-15 
> 22:45:20","policy":"","exclusive_consumer_tag":"","messages_ready":0,"messages_unacknowledged":0,"messages":0,"consumers":4,"active_consumers":4,"backing_queue_status":{"q1":0,"q2":0,"delta":["delta","undefined",0,"undefined"],"q3":0,"q4":0,"len":0,"pending_acks":0,"target_ram_count":0,"ram_msg_count":0,"ram_ack_count":0,"next_seq_id":0,"persistent_count":0,"avg_ingress_rate":0.0,"avg_egress_rate":0.0,"avg_ack_ingress_rate":0.0,"avg_ack_egress_rate":0.0},"messages_details":{"rate":0,"interval":90165823981,"last_event":1366065920939},"messages_ready_details":{"rate":0,"interval":90165823981,"last_event":1366065920939},"messages_unacknowledged_details":{"rate":0,"interval":90165823981,"last_event":1366065920939},"name":"Ingestion.1899vw88el","vhost":"/segment","durable":true,"auto_delete":false,"arguments":{},"node":"rabbit at rabbit2"}
>
> A few things to note: we have 4 servers subscribing to every queue, and 
> because of the RMQ client (node-amqp) we're using, it creates a channel per 
> subscription. I realize that's not ideal and we'll be fixing the client to 
> make multiple subscriptions per channel, but either way, the broker reports 
> only 350 MB of the memory use coming from the connections vs. 4.8 GB for 
> queues. 
>
> The broker has also been up for 76 days without downtime, so maybe there's 
> some kind of memory leak? We're seeing the memory go up over time. 
>
> Thanks!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130612/54820f41/attachment-0001.htm>