[rabbitmq-discuss] Disk/Memory Usage with RabbitMQ

[Barry Pederson]

Matthias Radestock wrote:

> How are you consuming messages from the queues? Delivered messages 
> (persistent or not) need to be acknowledged explicitly with basic.ack 
> unless the noAck flag for basic.get/basic.consume was set to true.

Yes, I've been trying both basic.ack and the noAck flag - so that part 
sounds OK.




> Also, what happens when you do one of the following:
> - close and re-open the channel on which messages have been consumed
> - close and re-open the connection on which messages have been consumed
> - restart the server
> and then start consuming messages from the same queues? Do you get 
> messages back that you have received before? If so that would be an 
> indication that you are not sending acknowledgments.

Haven't seen any repeated messages, so I guess that messages that were 
consumed were acked.  For the most part though, I don't have any queues 
at all bound to the exchange to which the persistent messages are being 
published.

After restarting the server just now, memory usage was back down, and 
the rabbit_persister.LOG file was fairly small (62k), where the 
rabbit_persister.LOG.previous is about 2.1mb

I'm fairly certain though that on another occasion a week or so ago, a 
restarted server came back up with large memory usage and a big 
rabbit_persister.LOG - I ended up wiping out the mnesia directory to get 
things back to square one.




>> Are there any other Erlang commands for examining the system to see
>> where any logjams may be?
> 
> Try
> 
> [process_info(P) || P <- processes(),
>                     process_info(P, memory) > {memory, 100000}].
> 
> which will list the details of all processes with a memory consumption 
> greater than 100000 bytes.


On first startup, the above bit just returns []

trying again after things grow for a while results in

------------------
[[{registered_name,rabbit_gc_persist},
   {current_function,{gen_server,loop,6}},
   {initial_call,{proc_lib,init_p,5}},
   {status,waiting},
   {message_queue_len,0},
   {messages,[]},
   {links,[<0.168.0>]},
   {dictionary,[{'$ancestors',[<0.168.0>,rabbit_sup,<0.103.0>]},
                {'$initial_call',{gen,init_it,
                                      [gen_server,
                                       <0.168.0>,
                                       <0.168.0>,
                                       {local,rabbit_gc_persist},
                                       rabbit_gc_persist,
                                       [],
                                       []]}}]},
   {trap_exit,false},
   {error_handler,error_handler},
   {priority,normal},
   {group_leader,<0.102.0>},
   {heap_size,121393},
   {stack_size,12},
   {reductions,74294},
   {garbage_collection,[{fullsweep_after,65535}]}]]
-------------------

And somewhat later on

--------------------
[[{registered_name,rabbit_gc_persist},
   {current_function,{gen_server,loop,6}},
   {initial_call,{proc_lib,init_p,5}},
   {status,waiting},
   {message_queue_len,0},
   {messages,[]},
   {links,[<0.168.0>]},
   {dictionary,[{'$ancestors',[<0.168.0>,rabbit_sup,<0.103.0>]},
                {'$initial_call',{gen,init_it,
                                      [gen_server,
                                       <0.168.0>,
                                       <0.168.0>,
                                       {local,rabbit_gc_persist},
                                       rabbit_gc_persist,
                                       [],
                                       []]}}]},
   {trap_exit,false},
   {error_handler,error_handler},
   {priority,normal},
   {group_leader,<0.102.0>},
   {heap_size,514229},
   {stack_size,12},
   {reductions,118305},
   {garbage_collection,[{fullsweep_after,65535}]}]]
----------------------

Looks like "heap_size" and "reductions" are growing.




>> Would publishing persistent messages to a durable exchange with no 
>> queues bound to it at all (yet) cause the messages to stick around?
> 
> You might be on to something there. My own testing shows a surprising 
> growth in memory and the persister log in this scenario. I will do some 
> more digging.

Ah good, I'm glad it's not just me.  Not knowing much Erlang I'm at a 
bit of a loss when it comes to troubleshooting this.

	Barry