[rabbitmq-discuss] RabbmitMQ consuming CPU and Memory

Paul Gross pgross at gmail.com
Wed May 30 16:00:47 BST 2012


>>>
>>> Unfortunately, by the time I was able to ssh into the server to run
>>> rabbitmqctl report, it said nodedown.
>>
>> That's a shame. Would be really good to catch it early enough to 
>> still get some useful output from rabbitmqctl.
>>
>
We set up a cron job to run "rabbitmqctl report" every 5 minutes. Here 
is the last report before CPU and memory spiked:

Status of node 'rabbit at bt-ci-rabbitmq' ...
[{pid,9152},
  {running_applications,[{rabbit,"RabbitMQ","2.8.2"},
                         {os_mon,"CPO  CXC 138 46","2.2.9"},
                         {sasl,"SASL  CXC 138 11","2.2.1"},
                         {mnesia,"MNESIA  CXC 138 12","4.7"},
                         {stdlib,"ERTS  CXC 138 10","1.18.1"},
                         {kernel,"ERTS  CXC 138 10","2.15.1"}]},
  {os,{unix,linux}},
  {erlang_version,"Erlang R15B01 (erts-5.9.1) [source] [64-bit] 
[smp:4:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
  {memory,[{total,1913089696},
           {processes,1890368996},
           {processes_used,1890265582},
           {system,22720700},
           {atom,586345},
           {atom_used,566016},
           {binary,5413240},
           {code,11905478},
           {ets,1274712}]},
  {vm_memory_high_watermark,0.19999999923056544},
  {vm_memory_limit,207944908},
  {disk_free_limit,1039724544},
  {disk_free,38008365056},
  {file_descriptors,[{total_limit,924},
                     {total_used,10},
                     {sockets_limit,829},
                     {sockets_used,6}]},
  {processes,[{limit,1048576},{used,787}]},
  {run_queue,261},
  {uptime,370840}]

Cluster status of node 'rabbit at bt-ci-rabbitmq' ...
[{nodes,[{disc,['rabbit at bt-ci-rabbitmq']}]},
  {running_nodes,['rabbit at bt-ci-rabbitmq']}]

Application environment of node 'rabbit at bt-ci-rabbitmq' ...
[{auth_backends,[rabbit_auth_backend_internal]},
  {auth_mechanisms,['PLAIN','AMQPLAIN']},
  {backing_queue_module,rabbit_variable_queue},
  {cluster_nodes,[]},
  {collect_statistics,none},
  {collect_statistics_interval,5000},
  {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
  {default_user,<<"guest">>},
  {default_user_tags,[administrator]},
  {default_vhost,<<"/">>},
  {delegate_count,16},
{disk_free_limit,{mem_relative,1.0}},
  {error_logger,{file,"/var/log/rabbitmq/rabbit at bt-ci-rabbitmq.log"}},
  {frame_max,131072},
  {hipe_compile,false},
  {included_applications,[]},
  {log_levels,[{connection,info}]},
  {msg_store_file_size_limit,16777216},
  {msg_store_index_module,rabbit_msg_store_ets_index},
  {queue_index_max_journal_entries,262144},
  {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at bt-ci-rabbitmq-sasl.log"}},
  {server_properties,[]},
  {ssl_cert_login_from,distinguished_name},
  {ssl_listeners,[]},
  {ssl_options,[]},
  {tcp_listen_options,[binary,
                       {packet,raw},
                       {reuseaddr,true},
                       {backlog,128},
                       {nodelay,true},
                       {linger,{true,0}},
                       {exit_on_close,false}]},
  {tcp_listeners,[5672]},
  {trace_vhosts,[]},
  {vm_memory_high_watermark,0.2}]

Connections:
pid     name    address port    peer_address    peer_port       ssl     
peer_cert_subject       peer_cert_issuer        peer_cert_validity      
auth_mechanism  ssl_protocol    ssl_key_exchange        ssl_cipher      
ssl_hash        protocol        user    vhost   timeout frame_max       
client_properties       recv_oct        recv_cnt        send_oct        
send_cnt        send_pend       state   last_blocked_by 
last_blocked_age        channels
<'rabbit at bt-ci-rabbitmq'.3.28408.1>     173.230.134.179:50847 -> 
50.116.38.166:5672     50.116.38.166   5672    173.230.134.179 50847   
false                           AMQPLAIN
                 {0,8,0} guest   /       0       131072  
[{"version","0.7.9"},{"platform","Ruby"},{"product","Bunny"},{"information","http://github.com/ruby-amqp/bunny"}]       
3805    7       346     6
        0       blocked resource        77.960635       1
<'rabbit at bt-ci-rabbitmq'.3.28432.1>     173.230.134.179:50962 -> 
50.116.38.166:5672     50.116.38.166   5672    173.230.134.179 50962   
false                           AMQPLAIN
                 {0,8,0} guest   /       0       131072  
[{"version","0.7.9"},{"platform","Ruby"},{"product","Bunny"},{"information","http://github.com/ruby-amqp/bunny"}]       
345     7       346     6
        0       blocked resource        63.68709        1
<'rabbit at bt-ci-rabbitmq'.3.28416.1>     173.230.134.179:50945 -> 
50.116.38.166:5672     50.116.38.166   5672    173.230.134.179 50945   
false                           AMQPLAIN
                 {0,8,0} guest   /       0       131072  
[{"version","0.7.9"},{"platform","Ruby"},{"product","Bunny"},{"information","http://github.com/ruby-amqp/bunny"}]       
345     7       346     6
        0       blocked resource        65.714869       1
<'rabbit at bt-ci-rabbitmq'.3.28400.1>     173.230.134.179:50817 -> 
50.116.38.166:5672     50.116.38.166   5672    173.230.134.179 50817   
false                           AMQPLAIN
                 {0,8,0} guest   /       0       131072  
[{"version","0.7.9"},{"platform","Ruby"},{"product","Bunny"},{"information","http://github.com/ruby-amqp/bunny"}]       
3805    7       346     6
        0       blocked resource        79.666493       1
<'rabbit at bt-ci-rabbitmq'.3.28424.1>     173.230.134.179:50949 -> 
50.116.38.166:5672     50.116.38.166   5672    173.230.134.179 50949   
false                           AMQPLAIN
                 {0,8,0} guest   /       0       131072  
[{"version","0.7.9"},{"platform","Ruby"},{"product","Bunny"},{"information","http://github.com/ruby-amqp/bunny"}]       
345     7       346     6
        0       blocked resource        64.824581       1

Channels:
pid     name    connection      number  user    vhost   transactional   
confirm consumer_count  messages_unacknowledged messages_unconfirmed    
messages_uncommitted    acks_uncommitted        prefetch_count  
client_flow_blocked
<'rabbit at bt-ci-rabbitmq'.3.28404.1>     173.230.134.179:50817 -> 
50.116.38.166:5672 (1) <'rabbit at bt-ci-rabbitmq'.3.28400.1>     1       
guest   /       false   false   0       0       0       0       0
        0       false
<'rabbit at bt-ci-rabbitmq'.3.28412.1>     173.230.134.179:50847 -> 
50.116.38.166:5672 (1) <'rabbit at bt-ci-rabbitmq'.3.28408.1>     1       
guest   /       false   false   0       0       0       0       0
        0       false
<'rabbit at bt-ci-rabbitmq'.3.28420.1>     173.230.134.179:50945 -> 
50.116.38.166:5672 (1) <'rabbit at bt-ci-rabbitmq'.3.28416.1>     1       
guest   /       false   false   0       0       0       0       0
        0       false
<'rabbit at bt-ci-rabbitmq'.3.28428.1>     173.230.134.179:50949 -> 
50.116.38.166:5672 (1) <'rabbit at bt-ci-rabbitmq'.3.28424.1>     1       
guest   /       false   false   0       0       0       0       0
        0       false
<'rabbit at bt-ci-rabbitmq'.3.28436.1>     173.230.134.179:50962 -> 
50.116.38.166:5672 (1) <'rabbit at bt-ci-rabbitmq'.3.28432.1>     1       
guest   /       false   false   0       0       0       0       0
        0       false

Queues on /:
pid     name    durable auto_delete     arguments       owner_pid       
slave_pids      synchronised_slave_pids exclusive_consumer_pid  
exclusive_consumer_tag  messages_ready  messages_unacknowledged 
messages        consumers       memory  backing_queue_status
<'rabbit at bt-ci-rabbitmq'.3.26809.1>     gateway_master_test     true    
false   [{"x-ha-policy","all"}]         []      []                      
131     0       131     0       143008  [{q1,0}, {q2,0}, 
{delta,{delta,undefined,0,undefined}}, {q3,131}, {q4,0}, {len,131}, 
{pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, 
{ram_ack_count,0}, {next_seq_id,131}, {persistent_count,131}, 
{avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, 
{avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}, {mirror_seen,0}, 
{mirror_senders,0}]
<'rabbit at bt-ci-rabbitmq'.3.19773.1>     gateway_release_test    true    
false   [{"x-ha-policy","all"}]         []      []                      
59      0       59      0       88888   [{q1,0}, {q2,0}, 
{delta,{delta,undefined,0,undefined}}, {q3,59}, {q4,0}, {len,59}, 
{pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, 
{ram_ack_count,0}, {next_seq_id,59}, {persistent_count,59}, 
{avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, 
{avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}, {mirror_seen,0}, 
{mirror_senders,0}]

Exchanges on /:
name    type    durable auto_delete     internal        arguments
amq.direct      direct  true    false   false   []
amq.topic       topic   true    false   false   []
amq.rabbitmq.trace      topic   true    false   false   []
amq.rabbitmq.log        topic   true    false   false   []
amq.fanout      fanout  true    false   false   []
amq.headers     headers true    false   false   []
         direct  true    false   false   []
amq.match       headers true    false   false   []

Bindings on /:
source_name     source_kind     destination_name        
destination_kind        routing_key     arguments
         exchange        gateway_master_test     queue   
gateway_master_test     []
         exchange        gateway_release_test    queue   
gateway_release_test    []

Consumers on /:

Permissions on /:
user    configure       write   read
guest   .*      .*      .*

End of server status report
...done.



More information about the rabbitmq-discuss mailing list