[rabbitmq-discuss] Memory usage

[Rob Harrop]

Hagbard Celine wrote:
> Thanks for your answer and sorry for the questions - i just want to know if RabbitMQ is a possible solution for my problem.
>
> --
>
> Does it mean that RabbitMQ will nerver hit the watermark if the producer slow enough (and the speed of my producers depends on the amount of memory i use)?

RabbitMQ will only hit the watermark if there are too many messages in 
memory. This can be caused by many different factors, the most likely 
being that messages are arriving at the broker faster than they can be 
sent to consumers or written to disk.

Rabbit will flush messages to disk if memory is low, but it also has to 
stop messages arriving while it does this otherwise it could still run 
out of memory while doing the paging.

Messages are only flushed to disk once the high watermark is hit, so 
you'll always see memory tend towards the watermark if the message 
ingress rate outweighs the message egress rate.

Once the high watermark is hit, messages are paged to disk until enough 
memory is reclaimed to drop below the watermark. For the duration of 
this process, producers are throttled.

>
> In my tests (with no consumers) the memory usage grows constantly with the amount of messages. Sometimes the memory usage decreased but overall it grows until the watermark. Are the producers still too fast for RabbitMQ (and my hardware)?

This is expected due to the behaviour above. If you have no consumers 
the only way to reclaim memory is to flush messages to disk.
>
> To solve my problem i have to guarantee that all messages will be queued and not blocked by RabbitMQ.

You have a few options I think, but the most sensible one is to simply 
perform your own throttling of your producer.

>
> Thanks a lot
> Hagbard
>
>> Hagbard,
>>
>> Since 2.x, RabbitMQ will no longer allow the Erlang process to run out
>> of memory and stop due to fast producers and slow consumers.
>>
>> If you have a fast producer running constantly, accompanied by a slow
>> consumer then the number of queued messages will grow. To prevent this
>> from causing out of memory problems, Rabbit will throttle the producer
>> once the memory high watermark is hit.
>>
>> The throttling works by simply blocking the producer from sending any
>> more data over the TCP socket. Once memory drops below the high
>> watermark, due to the consumer catching up or messages being flushed to
>> disk, the producer can send data again.
>>
>> Note that your producer might bounce off the watermark frequently if it
>> just keeps publishing without let up.
>>
>> Rob
>>
>> Hagbard at gmx.de wrote:
>>> Hi,
>>> im new to RabbitMQ and i have a question about the memory usage of
>> RabbitMQ. I read that since RabbitMQ 2.x i don't need to care about fast
>> producers and slow consumers and so don't need to care about the amount of
>> messages in a queue and memory usage. After a test yesterday with 2GB memory and
>> 4M messages in a queue (no consumer) RabbitMQ reached high memory watermark
>> and stopped. Can somebody explain that behaviour?
>>> Thanks a lot
>>> Hagbard
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>