[rabbitmq-discuss] Cluster Memory Usage

Mon Nov 21 21:41:12 GMT 2011

Travis,

On 21/11/11 21:11, Travis wrote:
>> How many messages were in the queue at the time?
>
> The queue was sitting at about 10 thousand messages.

Have you ever seen the excessive memory use when there were no / very 
few messages in the queue?

Also, what CPU usage do you see when memory use is excessive? On the 
slave you are probably never going to be able to utilise much more than 
one core, since there is only one queue and no connections. So if CPU 
usage is getting near maxing out one core then the system will be 
operating at the limit.

>> Also, what memory limit does rabbit report in its log?
>
> Memory limit set to 2408MB.
>
> We have the vm_memory_high_watermark set to 10%.  The system has 24GB
> of memory.

The reason I was asking for what gets reported in the logs is just to 
make sure the settings have taken effect.

>> You mentioned that publishers use 'confirms'. Do they wait for confirmation
>> after every message, after some batch of messages, or do they let confirms
>> stream in completely independently?
>
> We're shoveling from remote rabbitmqs into the cluster.  The shovels have a
> prefetch of 1000.  They are set to confirm.  So I presume that the answer to
> your questions depends on how the shovel works.

The shovel simply converts confirms from the destination to acks on the 
source. So in your setup, due to the prefetch on the source, publishing 
will pause if there are 1000 confirms pending.

Regards,

Matthias.