[rabbitmq-discuss] why rabbit eating so much memory?
liangrubo at bytedance.com
Thu Jul 11 02:35:53 BST 2013
There is no other vhosts. I can't keep the problem for investigation since
it is a production server. "rabbitmqctl stop_app; rabbitmqctl start_app"
frees the memory.
I am new to rabbit and this is just one of a series problems I meet. I am
not sure whether it helps but I am going to list the problems I experienced
to provide more context.
- Debian 7.1
- A cluster with 3 servers
- A virtual ip is assigned to the servers, using keepalive to shift the
- use default exchange
- 6 queues (ha.snssdk.log, ha.snssdk.log.dispatch,
session.log, session.output), queues whose name begin with 'ha.' are
- all queues and messages are none-durable
- message publish rate: 4000 ~ 10000 msg/s
Problems(with actions I took):
1. Initially use rabbit version 3.1.1, the rabbit server crash with message
like following sometime: Slogan: std_alloc: Cannot allocate
1119579832804544 bytes of memory (of type "arg_reg").
- action: upgrade both rabbit and erlang to the latest version.
2. rabbit take up to much memory
- action: rabbitmqctl stop_app; rabbitmqctl start_app
3. block on declaring queue 'ha.snssdk.log.dispatch', but it is OK to
declare other queues, including 'ha.snssdk.log.dispatch2'
- action: stop rabbit, remove mnesia folder, start rabbit and rebuild
4. the management_plugin doesn't work, with message like "TypeError: Cannot
read property 'connections' of undefined"
- action: asking in maillist but no action is taken so far
5. though all the queues and messages are none-durable, and the memory
consumption is far below configured by 'vm_memory_high_watermark', the
mnesia folder sometimes can take up much space(e.g. 19G), it is basically
taken by 'msg_store_transient' folder. I don't understand why.
- action: none
6. sometimes rabbit doesn't push message to consumers in time: I found some
consumers don't get message from time to time even there are messages in
the queue. But this seems to be transient (lasting for about 1 minute).
- action: none
On Thu, Jul 11, 2013 at 12:55 AM, Matthias Radestock
<matthias at rabbitmq.com>wrote:
> On 09/07/13 15:59, Rubo Liang wrote:
>> My rabbit eats 25G memory, but the queues don't take to much as shown by
>> this command: 'rabbitmqctl list_queues name messages memory':
>> Listing queues ...
>> ha.snssdk.log 695954 16426216
>> ha.snssdk.log.dispatch 660748 102698808
>> ha.snssdk.log.output 63573 32800680
>> ssesion 0 25952
>> ssesion.log 101 436328
>> ssesion.output 0 148816
>> As shown above the queues only take about 150M memory. But the rabbit
>> process actually takes about 25G.
>> 'rabbitmqctl status' gives me following output on memory:
> So that shows queue processes taking up 19G. Perhaps you have queue's in
> vhosts other than '/'? Note that 'rabbitmqctl list_queues' only shows
> information on one vhost ('/' by default).
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss