[rabbitmq-discuss] memory usage

[Philip Stubbings]

Hi Matthias, Thank you for your reply.

I used rabbitmqctl to determine the status of my queues and my messages are flagged as persistent. 
I have run the same test again, but this time I inject/consume a few messages later on, as you suggested. 
However, the memory footprint remains the same:


After injecting 1 million messages
----------------------------------

./rabbitmqctl list_queues -p / name durable auto_delete messages_ready messages_unacknowledged memory consumers
Listing queues ...
myQueue 	true    false   1000000 0       1324422960      0
...done.

top:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11243 phil      23   0 3480m 3.0g 2048 S    0 39.0  10:56.64 beam.smp

persister logs:
-rw-rw-r--  1 phil phil 286M Feb  9 11:43 rabbit_persister.LOG
-rw-rw-r--  1 phil phil 196M Feb  9 11:42 rabbit_persister.LOG.previous
 
After consuming 1 million messages with a single consumer
---------------------------------------------------------

rabbitmqctl list_queues -p / name durable auto_delete messages_ready messages_unacknowledged memory consumers
Listing queues ...
myExchange      false   false   0       0       3976    0
myQueue 	true    false   0       0       3976    0
...done.

top:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11243 phil      23   0 2903m 1.8g 2048 S    0 23.5  21:55.42 beam.smp

persister logs:
-rw-rw-r--  1 phil phil 463M Feb  9 12:00 rabbit_persister.LOG
-rw-rw-r--  1 phil phil 303M Feb  9 12:00 rabbit_persister.LOG.previous

After injecting/consuming a few messages (after consuming the first million), beam's use of physical 
memory (1.8G) remains the same... Moreover, should the persister logs be this large after consuming 
(and acknowledging) all messages? 

I intend to use rabbit in a SMS routing application, where typically I need to persist messages at a sustained rate 
of 30 - 90 messages per second. Less frequently, there will be bursts of 1 million or more messages.

Thanks again,

Philip


On Fri, Feb 06, 2009 at 06:19:11PM +0000, Matthias Radestock wrote:
> Philip,
>
> Philip Stubbings wrote:
>> I have set up Rabbit 1.5.1 running as a single node. During testing, I
>> notice that memory use is extremely high. More specifically, if I push
>> 1 million messages to a queue and later consume them (with
>> acknowledgment) memory usage remains very high (in excess of 3G). I can
>> verify that all consumed messages where acknowledged and that the queue
>> is empty. Why does the process still require 3G of memory even after 24
>> hours of inactivity? 
>
> How did you determine that the queues were empty?
>
> Are your messages flagged as persistent? How big is the persister log  
> (usually in /var/lib/rabbitmq/mnesia/rabbit/rabbit_persister.LOG).
>
> Can you send us the code to reproduce the problem?
>
>> The only way to reduce the memory footprint is to restart rabbitmq.,
>> However this is not an option for me.
>
> Erlang processes only perform garbage collection when there is some  
> activity. That usually isn't a problem, since most systems are not  
> expected to be both idle for a long time and have other processes  
> needing memory at the same time.
>
> Does the memory usage remain high when you inject & consume some more  
> messages?
>
>> The second problem I have encountered is during, and immediately after
>> pushing 1 million messages to rabbitmq. If I try to connect a consumer
>> whilst messages are being produced, I get the following error on the
>> consumer side:
>>
>> (connection error; reason:
>> {#method<connection.close>(reply-code=541,reply-text=INTERNAL_ERROR,class-id=0,method-id=0),null,""})
>>
>> While, on the rabbit server:
>> =ERROR REPORT==== 5-Feb-2009::12:43:57 ===
>> connection <0.3155.0> (running), channel 1 - error:
>> {timeout,{gen_server,call,[rabbit_persister,serial]}}
>> =WARNING REPORT==== 5-Feb-2009::12:43:57 ===
>> Non-AMQP exit reason
>> '{timeout,{gen_server,call,[rabbit_persister,serial]}}'
>
> That looks like the same problem Darien Kindlund reported. See my  
> response to him a few moments ago.
>
>> And also, during and just after the 1 million messages have been
>> produced (with no consumers connected) I get the following error while
>> trying to examine the queue status:
>>
>> ../rabbitmq-server-1.5.1/scripts/rabbitmqctl list_queues -p /
>> Listing queues ...
>> Error: {badrpc,
>>            {'EXIT',
>> 	        {timeout,
>> 			{gen_server,call,[<4472.2308.0>,{info,[name,messages]}]}}}}
>
> The queues are probably busy processing the message backlog. The info  
> request will be stuck behind the messages. I have filed a bug to do  
> something about that.
>
> Is the kind of bursty behaviour you are testing - with lots of messages  
> being injected at a high rate followed by periods of inactivity -  
> typical of your application?
>
>
> Regards,
>
> Matthias.

--