[rabbitmq-discuss] RabbmitMQ consuming CPU and Memory

[Paul Gross]

We are seeing weird issues with RabbmitMQ and CPU/Memory. Basically, we 
run tests locally and on our build server. In both cases, we create a 
couple of queues, push messages through (not many, maybe a few hundred), 
and consume them. This works fine for a few days.

Then, all of a sudden, RabbitMQ goes nuts. It consumes all available CPU 
and memory until it crashes the server. As far as we can tell, nothing 
is different right before this happens. We have to manually kill 
RabbitMQ, and when it comes back up, the queue has less than 10 
messages. The last time this happened, the only thing in the log was 
"Mnesia is overloaded."

This happens both locally on a MacPro (10.7.3) and on our build server 
(Ubuntu 10.04).

These images are from munin, and give an idea of what's going on:

http://dl.dropbox.com/u/15440955/rabbitmq-cpu.png
http://dl.dropbox.com/u/15440955/rabbitmq-memory.png

We are looking for help on how to debug this problem. We cannot directly 
reproduce the problem, but it happens every couple of days.

Here is our setup on the build server:

    $ sudo rabbitmqctl status
    Status of node 'rabbit at ci-rabbitmq' ...
    [{pid,31302},
      {running_applications,[{rabbit,"RabbitMQ","2.7.1"},
                             {os_mon,"CPO  CXC 138 46","2.2.4"},
                             {sasl,"SASL  CXC 138 11","2.1.8"},
                             {mnesia,"MNESIA  CXC 138 12","4.4.12"},
                             {stdlib,"ERTS  CXC 138 10","1.16.4"},
                             {kernel,"ERTS  CXC 138 10","2.13.4"}]},
      {os,{unix,linux}},
      {erlang_version,"Erlang R13B03 (erts-5.7.4) [source] [64-bit]
    [smp:4:4] [rq:4] [async-threads:30] [h
    ipe] [kernel-poll:true]\n"},
      {memory,[{total,40501088},
               {processes,15379528},
               {processes_used,15355904},
               {system,25121560},
               {atom,1113129},
               {atom_used,1103706},
               {binary,8548768},
               {code,11903849},
               {ets,1436704}]},
      {vm_memory_high_watermark,0.19999999923056544},
      {vm_memory_limit,207944908}]
    ...done.


And here is our setup on the MacPro:

    Status of node rabbit at yoshi ...
    [{pid,70356},
      {running_applications,[{rabbit,"RabbitMQ","2.7.1"},
                             {mnesia,"MNESIA  CXC 138 12","4.6"},
                             {os_mon,"CPO  CXC 138 46","2.2.8"},
                             {sasl,"SASL  CXC 138 11","2.2"},
                             {stdlib,"ERTS  CXC 138 10","1.18"},
                             {kernel,"ERTS  CXC 138 10","2.15"}]},
      {os,{unix,darwin}},
      {erlang_version,"Erlang R15B (erts-5.9) [source] [64-bit]
    [smp:16:16] [async-threads:30] [hipe] [kernel-poll:true]\n"},
      {memory,[{total,39776688},
               {processes,13068508},
               {processes_used,13068480},
               {system,26708180},
               {atom,504409},
               {atom_used,475853},
               {binary,8659336},
               {code,11916223},
               {ets,1464712}]},
      {vm_memory_high_watermark,0.3999999999026046},
      {vm_memory_limit,3285576908}]
    ...done.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120510/08ba4552/attachment.htm>