[rabbitmq-discuss] does rabbit eat memory?

Fri Nov 26 16:44:51 GMT 2010

On 26 November 2010 16:33, Matthew Sackman <matthew at rabbitmq.com> wrote:
> On Fri, Nov 26, 2010 at 04:28:14PM +0000, Robert Fuller wrote:
>> On 26 November 2010 16:19, Matthew Sackman <matthew at rabbitmq.com> wrote:
>> > On Fri, Nov 26, 2010 at 04:15:53PM +0000, Robert Fuller wrote:
>> >> Having noticed memory consumption on our server I ran a test of using
>> >> a clean install, and adding 3600 messages to the queue, then consuming
>> >> these messages and checking for memory usage using top command. I
>> >> notice that the memory used by rabbitmq slowly creeps upwards.
>> >
>> > Are you acknowledging messages or consuming with noAck/autoAck = true ?
>>
>> The consumer is acknowledging each message.
>
> Ok, in which case there's nothing more you can do. Erlang is a GC'd
> language and so when it obtains memory on demand, it may choose not to
> promptly return that memory to the OS. It might take a while for some
> large contiguous block of free memory to occur before Erlang free's the
> memory.
>
> Thus whilst the memory use of Rabbit may initially rise, provided you
> keep draining the queues such that they don't grow endlessly, you should
> find the memory use levels out after a while.

Thanks Mathew, this makes sense and also maybe sheds some light on the
larger picture which is this: Over a number of hours during our
producers were producing a few million messages at about the same rate
the consumer was able to process them. However, because the consumer
started after the producer, there remained at all times some thousands
of messages in the queue. Memory consumption seemed to rise and rise.
Is it only when the queue reaches a zero size that the memory is
cleared?

What strategies would you suggest for dealing with this?

One I can think of would be to maintain a rotation of queues... after
putting a million messages onto any queue move on to the next queue.
The consumer, finding no messages on a queue will move to the next
queue in the rotation. Is this reasonable?

>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>