[rabbitmq-discuss] RabbmitMQ consuming CPU and Memory

Fri May 18 18:36:15 BST 2012

> That looks like a perfectly healthy rabbit.
>
> There are no user-defined exchanges, queues or bindings - is that 
> expected?
>
This server is running a build, which creates and deletes a couple of 
queues. It's possible that these creates/deletes are happening at the 
same time. Could this cause the observed behavior?

> In your original email you said that "all of a sudden [...] RabbitMQ 
> consumes all available CPU and memory", but in the report the memory 
> usage is shown as 90MB, which isn't very high.
>
> Are you sure that it is *rabbit* that is eating all the CPU & memory? 
> Please check the CPU/memory usage of the beam.smp process and include 
> that in your report next time the problem occurs.
>
The beam.smp process is definitely going nuts. Here's the output from ps 
aufx:

rabbitmq 26472  0.0  0.0   4144     0 ?        S    May14   0:00 /bin/sh 
/usr/sbin/rabbitmq-server
rabbitmq 26490  184 82.4 2367176 837356 ?      Sl   May14 9852:39  \_ 
/usr/lib/erlang/erts-5.9.1/bin/beam.smp -W w -K true -A30 -P 1048576 -- 
-root /usr/lib/erlang -progname erl -- -
rabbitmq 26605  0.0  0.0  10776     0 ?        Ss   May14   0:00      \_ 
inet_gethost 4
rabbitmq 26606  0.0  0.0  12876    36 ?        S    May14   
0:00          \_ inet_gethost 4

As you can see, beam.smp is using 184% CPU and 82.4% of memory on this 
server (4 cores, 1GB RAM).

Unfortunately, by the time I was able to ssh into the server to run 
rabbitmqctl report, it said nodedown.

Here's the output of report after I manually killed beam.smp and ran 
/etc/init.d/rabbitmq start:

Status of node 'rabbit at bt-ci-rabbitmq' ...
[{pid,18399},
  {running_applications,[{rabbit,"RabbitMQ","2.8.2"},
                         {os_mon,"CPO  CXC 138 46","2.2.9"},
                         {sasl,"SASL  CXC 138 11","2.2.1"},
                         {mnesia,"MNESIA  CXC 138 12","4.7"},
                         {stdlib,"ERTS  CXC 138 10","1.18.1"},
                         {kernel,"ERTS  CXC 138 10","2.15.1"}]},
  {os,{unix,linux}},
  {erlang_version,"Erlang R15B01 (erts-5.9.1) [source] [64-bit] 
[smp:4:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
  {memory,[{total,27528008},
           {processes,10447908},
           {processes_used,10447894},
           {system,17080100},
           {atom,504409},
           {atom_used,475802},
           {binary,806272},
           {code,11905478},
           {ets,823624}]},
  {vm_memory_high_watermark,0.19999999923056544},
  {vm_memory_limit,207944908},
  {disk_free_limit,1039724544},
  {disk_free,38120873984},
  {file_descriptors,[{total_limit,924},
                     {total_used,6},
                     {sockets_limit,829},
                     {sockets_used,1}]},
  {processes,[{limit,1048576},{used,130}]},
  {run_queue,0},
  {uptime,27}]

Cluster status of node 'rabbit at bt-ci-rabbitmq' ...
[{nodes,[{disc,['rabbit at bt-ci-rabbitmq']}]},
  {running_nodes,['rabbit at bt-ci-rabbitmq']}]

Application environment of node 'rabbit at bt-ci-rabbitmq' ...
[{auth_backends,[rabbit_auth_backend_internal]},
  {auth_mechanisms,['PLAIN','AMQPLAIN']},
  {backing_queue_module,rabbit_variable_queue},
  {cluster_nodes,[]},
  {collect_statistics,none},
  {collect_statistics_interval,5000},
  {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
  {default_user,<<"guest">>},
  {default_user_tags,[administrator]},
  {default_vhost,<<"/">>},
  {delegate_count,16},
  {disk_free_limit,{mem_relative,1.0}},
  {error_logger,{file,"/var/log/rabbitmq/rabbit at bt-ci-rabbitmq.log"}},
  {frame_max,131072},
  {hipe_compile,false},
  {included_applications,[]},
  {log_levels,[{connection,info}]},
  {msg_store_file_size_limit,16777216},
  {msg_store_index_module,rabbit_msg_store_ets_index},
  {queue_index_max_journal_entries,262144},
  {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at bt-ci-rabbitmq-sasl.log"}},
  {server_properties,[]},
  {ssl_cert_login_from,distinguished_name},
  {ssl_listeners,[]},
  {ssl_options,[]},
  {tcp_listen_options,[binary,
                       {packet,raw},
                       {reuseaddr,true},
                       {backlog,128},
                       {nodelay,true},
                       {linger,{true,0}},
                       {exit_on_close,false}]},
  {tcp_listeners,[5672]},
  {trace_vhosts,[]},
  {vm_memory_high_watermark,0.2}]

Connections:

Channels:

Queues on /:
pid    name    durable    auto_delete    arguments    owner_pid    
slave_pids    synchronised_slave_pids    exclusive_consumer_pid    
exclusive_consumer_tag    messages_ready    messages_unacknowledged    
messages    consumers    memory    backing_queue_status
<'rabbit at bt-ci-rabbitmq'.2.186.0>    
webhook_worker_master-testing_notifications    true    false    
[{"x-ha-policy","all"}]        []    []            0    0    0    0    
34688    [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, 
{q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, 
{ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,0}, 
{persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, 
{avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}, {mirror_seen,0}, 
{mirror_senders,0}]
<'rabbit at bt-ci-rabbitmq'.2.187.0>    gateway_master_test    true    
false    [{"x-ha-policy","all"}]        []    []            79    0    
79    0    88808    [{q1,0}, {q2,0}, 
{delta,{delta,undefined,0,undefined}}, {q3,79}, {q4,0}, {len,79}, 
{pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, 
{ram_ack_count,0}, {next_seq_id,16384}, {persistent_count,79}, 
{avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, 
{avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}, {mirror_seen,0}, 
{mirror_senders,0}]
<'rabbit at bt-ci-rabbitmq'.2.188.0>    gateway_release_test    true    
false    [{"x-ha-policy","all"}]        []    []            109    0    
109    0    142928    [{q1,0}, {q2,0}, 
{delta,{delta,undefined,0,undefined}}, {q3,109}, {q4,0}, {len,109}, 
{pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, 
{ram_ack_count,0}, {next_seq_id,16384}, {persistent_count,109}, 
{avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, 
{avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}, {mirror_seen,0}, 
{mirror_senders,0}]

Exchanges on /:
name    type    durable    auto_delete    internal    arguments
amq.direct    direct    true    false    false    []
amq.topic    topic    true    false    false    []
amq.rabbitmq.trace    topic    true    false    false    []
amq.rabbitmq.log    topic    true    false    false    []
amq.fanout    fanout    true    false    false    []
amq.headers    headers    true    false    false    []
     direct    true    false    false    []
amq.match    headers    true    false    false    []

Bindings on /:
source_name    source_kind    destination_name    destination_kind    
routing_key    arguments
     exchange    gateway_master_test    queue    gateway_master_test    []
     exchange    gateway_release_test    queue    gateway_release_test    []
     exchange    webhook_worker_master-testing_notifications    queue    
webhook_worker_master-testing_notifications    []

Consumers on /:

Permissions on /:
user    configure    write    read
guest    .*    .*    .*

End of server status report
...done.