[rabbitmq-discuss] Memory usage

Rob Harrop rob at rabbitmq.com
Tue May 24 15:37:23 BST 2011



Rob Harrop wrote:
>
>
> Hagbard Celine wrote:
>> Thanks for your answer and sorry for the questions - i just want to
>> know if RabbitMQ is a possible solution for my problem.
>>
>> --
>>
>> Does it mean that RabbitMQ will nerver hit the watermark if the
>> producer slow enough (and the speed of my producers depends on the
>> amount of memory i use)?
>
> RabbitMQ will only hit the watermark if there are too many messages in
> memory. This can be caused by many different factors, the most likely
> being that messages are arriving at the broker faster than they can be
> sent to consumers or written to disk.
>
> Rabbit will flush messages to disk if memory is low, but it also has to
> stop messages arriving while it does this otherwise it could still run
> out of memory while doing the paging.
>
> Messages are only flushed to disk once the high watermark is hit, so
> you'll always see memory tend towards the watermark if the message
> ingress rate outweighs the message egress rate.

Of course, this is actually nonsense. What I wanted to write is that 
Rabbit will flush messages to disk as needed, even if the watermark 
isn't hit. However, if you do hit the watermark then Rabbit has to flush 
messages to disk *and* it can't receive any more - hence the blocking 
behaviour on the producers.

>
> Once the high watermark is hit, messages are paged to disk until enough
> memory is reclaimed to drop below the watermark. For the duration of
> this process, producers are throttled.
>
>>
>> In my tests (with no consumers) the memory usage grows constantly with
>> the amount of messages. Sometimes the memory usage decreased but
>> overall it grows until the watermark. Are the producers still too fast
>> for RabbitMQ (and my hardware)?
>
> This is expected due to the behaviour above. If you have no consumers
> the only way to reclaim memory is to flush messages to disk.
>>
>> To solve my problem i have to guarantee that all messages will be
>> queued and not blocked by RabbitMQ.
>
> You have a few options I think, but the most sensible one is to simply
> perform your own throttling of your producer.
>
>>
>> Thanks a lot
>> Hagbard
>>
>>> Hagbard,
>>>
>>> Since 2.x, RabbitMQ will no longer allow the Erlang process to run out
>>> of memory and stop due to fast producers and slow consumers.
>>>
>>> If you have a fast producer running constantly, accompanied by a slow
>>> consumer then the number of queued messages will grow. To prevent this
>>> from causing out of memory problems, Rabbit will throttle the producer
>>> once the memory high watermark is hit.
>>>
>>> The throttling works by simply blocking the producer from sending any
>>> more data over the TCP socket. Once memory drops below the high
>>> watermark, due to the consumer catching up or messages being flushed to
>>> disk, the producer can send data again.
>>>
>>> Note that your producer might bounce off the watermark frequently if it
>>> just keeps publishing without let up.
>>>
>>> Rob
>>>
>>> Hagbard at gmx.de wrote:
>>>> Hi,
>>>> im new to RabbitMQ and i have a question about the memory usage of
>>> RabbitMQ. I read that since RabbitMQ 2.x i don't need to care about fast
>>> producers and slow consumers and so don't need to care about the
>>> amount of
>>> messages in a queue and memory usage. After a test yesterday with 2GB
>>> memory and
>>> 4M messages in a queue (no consumer) RabbitMQ reached high memory
>>> watermark
>>> and stopped. Can somebody explain that behaviour?
>>>> Thanks a lot
>>>> Hagbard
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>


More information about the rabbitmq-discuss mailing list