[rabbitmq-discuss] RabbmitMQ consuming CPU and Memory
Paul Gross
pgross at gmail.com
Wed May 30 16:00:47 BST 2012
>>>
>>> Unfortunately, by the time I was able to ssh into the server to run
>>> rabbitmqctl report, it said nodedown.
>>
>> That's a shame. Would be really good to catch it early enough to
>> still get some useful output from rabbitmqctl.
>>
>
We set up a cron job to run "rabbitmqctl report" every 5 minutes. Here
is the last report before CPU and memory spiked:
Status of node 'rabbit at bt-ci-rabbitmq' ...
[{pid,9152},
{running_applications,[{rabbit,"RabbitMQ","2.8.2"},
{os_mon,"CPO CXC 138 46","2.2.9"},
{sasl,"SASL CXC 138 11","2.2.1"},
{mnesia,"MNESIA CXC 138 12","4.7"},
{stdlib,"ERTS CXC 138 10","1.18.1"},
{kernel,"ERTS CXC 138 10","2.15.1"}]},
{os,{unix,linux}},
{erlang_version,"Erlang R15B01 (erts-5.9.1) [source] [64-bit]
[smp:4:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,[{total,1913089696},
{processes,1890368996},
{processes_used,1890265582},
{system,22720700},
{atom,586345},
{atom_used,566016},
{binary,5413240},
{code,11905478},
{ets,1274712}]},
{vm_memory_high_watermark,0.19999999923056544},
{vm_memory_limit,207944908},
{disk_free_limit,1039724544},
{disk_free,38008365056},
{file_descriptors,[{total_limit,924},
{total_used,10},
{sockets_limit,829},
{sockets_used,6}]},
{processes,[{limit,1048576},{used,787}]},
{run_queue,261},
{uptime,370840}]
Cluster status of node 'rabbit at bt-ci-rabbitmq' ...
[{nodes,[{disc,['rabbit at bt-ci-rabbitmq']}]},
{running_nodes,['rabbit at bt-ci-rabbitmq']}]
Application environment of node 'rabbit at bt-ci-rabbitmq' ...
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue},
{cluster_nodes,[]},
{collect_statistics,none},
{collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,{mem_relative,1.0}},
{error_logger,{file,"/var/log/rabbitmq/rabbit at bt-ci-rabbitmq.log"}},
{frame_max,131072},
{hipe_compile,false},
{included_applications,[]},
{log_levels,[{connection,info}]},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{queue_index_max_journal_entries,262144},
{sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at bt-ci-rabbitmq-sasl.log"}},
{server_properties,[]},
{ssl_cert_login_from,distinguished_name},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,[binary,
{packet,raw},
{reuseaddr,true},
{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[5672]},
{trace_vhosts,[]},
{vm_memory_high_watermark,0.2}]
Connections:
pid name address port peer_address peer_port ssl
peer_cert_subject peer_cert_issuer peer_cert_validity
auth_mechanism ssl_protocol ssl_key_exchange ssl_cipher
ssl_hash protocol user vhost timeout frame_max
client_properties recv_oct recv_cnt send_oct
send_cnt send_pend state last_blocked_by
last_blocked_age channels
<'rabbit at bt-ci-rabbitmq'.3.28408.1> 173.230.134.179:50847 ->
50.116.38.166:5672 50.116.38.166 5672 173.230.134.179 50847
false AMQPLAIN
{0,8,0} guest / 0 131072
[{"version","0.7.9"},{"platform","Ruby"},{"product","Bunny"},{"information","http://github.com/ruby-amqp/bunny"}]
3805 7 346 6
0 blocked resource 77.960635 1
<'rabbit at bt-ci-rabbitmq'.3.28432.1> 173.230.134.179:50962 ->
50.116.38.166:5672 50.116.38.166 5672 173.230.134.179 50962
false AMQPLAIN
{0,8,0} guest / 0 131072
[{"version","0.7.9"},{"platform","Ruby"},{"product","Bunny"},{"information","http://github.com/ruby-amqp/bunny"}]
345 7 346 6
0 blocked resource 63.68709 1
<'rabbit at bt-ci-rabbitmq'.3.28416.1> 173.230.134.179:50945 ->
50.116.38.166:5672 50.116.38.166 5672 173.230.134.179 50945
false AMQPLAIN
{0,8,0} guest / 0 131072
[{"version","0.7.9"},{"platform","Ruby"},{"product","Bunny"},{"information","http://github.com/ruby-amqp/bunny"}]
345 7 346 6
0 blocked resource 65.714869 1
<'rabbit at bt-ci-rabbitmq'.3.28400.1> 173.230.134.179:50817 ->
50.116.38.166:5672 50.116.38.166 5672 173.230.134.179 50817
false AMQPLAIN
{0,8,0} guest / 0 131072
[{"version","0.7.9"},{"platform","Ruby"},{"product","Bunny"},{"information","http://github.com/ruby-amqp/bunny"}]
3805 7 346 6
0 blocked resource 79.666493 1
<'rabbit at bt-ci-rabbitmq'.3.28424.1> 173.230.134.179:50949 ->
50.116.38.166:5672 50.116.38.166 5672 173.230.134.179 50949
false AMQPLAIN
{0,8,0} guest / 0 131072
[{"version","0.7.9"},{"platform","Ruby"},{"product","Bunny"},{"information","http://github.com/ruby-amqp/bunny"}]
345 7 346 6
0 blocked resource 64.824581 1
Channels:
pid name connection number user vhost transactional
confirm consumer_count messages_unacknowledged messages_unconfirmed
messages_uncommitted acks_uncommitted prefetch_count
client_flow_blocked
<'rabbit at bt-ci-rabbitmq'.3.28404.1> 173.230.134.179:50817 ->
50.116.38.166:5672 (1) <'rabbit at bt-ci-rabbitmq'.3.28400.1> 1
guest / false false 0 0 0 0 0
0 false
<'rabbit at bt-ci-rabbitmq'.3.28412.1> 173.230.134.179:50847 ->
50.116.38.166:5672 (1) <'rabbit at bt-ci-rabbitmq'.3.28408.1> 1
guest / false false 0 0 0 0 0
0 false
<'rabbit at bt-ci-rabbitmq'.3.28420.1> 173.230.134.179:50945 ->
50.116.38.166:5672 (1) <'rabbit at bt-ci-rabbitmq'.3.28416.1> 1
guest / false false 0 0 0 0 0
0 false
<'rabbit at bt-ci-rabbitmq'.3.28428.1> 173.230.134.179:50949 ->
50.116.38.166:5672 (1) <'rabbit at bt-ci-rabbitmq'.3.28424.1> 1
guest / false false 0 0 0 0 0
0 false
<'rabbit at bt-ci-rabbitmq'.3.28436.1> 173.230.134.179:50962 ->
50.116.38.166:5672 (1) <'rabbit at bt-ci-rabbitmq'.3.28432.1> 1
guest / false false 0 0 0 0 0
0 false
Queues on /:
pid name durable auto_delete arguments owner_pid
slave_pids synchronised_slave_pids exclusive_consumer_pid
exclusive_consumer_tag messages_ready messages_unacknowledged
messages consumers memory backing_queue_status
<'rabbit at bt-ci-rabbitmq'.3.26809.1> gateway_master_test true
false [{"x-ha-policy","all"}] [] []
131 0 131 0 143008 [{q1,0}, {q2,0},
{delta,{delta,undefined,0,undefined}}, {q3,131}, {q4,0}, {len,131},
{pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0},
{ram_ack_count,0}, {next_seq_id,131}, {persistent_count,131},
{avg_ingress_rate,0.0}, {avg_egress_rate,0.0},
{avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}, {mirror_seen,0},
{mirror_senders,0}]
<'rabbit at bt-ci-rabbitmq'.3.19773.1> gateway_release_test true
false [{"x-ha-policy","all"}] [] []
59 0 59 0 88888 [{q1,0}, {q2,0},
{delta,{delta,undefined,0,undefined}}, {q3,59}, {q4,0}, {len,59},
{pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0},
{ram_ack_count,0}, {next_seq_id,59}, {persistent_count,59},
{avg_ingress_rate,0.0}, {avg_egress_rate,0.0},
{avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}, {mirror_seen,0},
{mirror_senders,0}]
Exchanges on /:
name type durable auto_delete internal arguments
amq.direct direct true false false []
amq.topic topic true false false []
amq.rabbitmq.trace topic true false false []
amq.rabbitmq.log topic true false false []
amq.fanout fanout true false false []
amq.headers headers true false false []
direct true false false []
amq.match headers true false false []
Bindings on /:
source_name source_kind destination_name
destination_kind routing_key arguments
exchange gateway_master_test queue
gateway_master_test []
exchange gateway_release_test queue
gateway_release_test []
Consumers on /:
Permissions on /:
user configure write read
guest .* .* .*
End of server status report
...done.
More information about the rabbitmq-discuss
mailing list