[rabbitmq-discuss] RabbmitMQ consuming CPU and Memory
Paul Gross
pgross at gmail.com
Fri May 18 18:36:15 BST 2012
> That looks like a perfectly healthy rabbit.
>
> There are no user-defined exchanges, queues or bindings - is that
> expected?
>
This server is running a build, which creates and deletes a couple of
queues. It's possible that these creates/deletes are happening at the
same time. Could this cause the observed behavior?
> In your original email you said that "all of a sudden [...] RabbitMQ
> consumes all available CPU and memory", but in the report the memory
> usage is shown as 90MB, which isn't very high.
>
> Are you sure that it is *rabbit* that is eating all the CPU & memory?
> Please check the CPU/memory usage of the beam.smp process and include
> that in your report next time the problem occurs.
>
The beam.smp process is definitely going nuts. Here's the output from ps
aufx:
rabbitmq 26472 0.0 0.0 4144 0 ? S May14 0:00 /bin/sh
/usr/sbin/rabbitmq-server
rabbitmq 26490 184 82.4 2367176 837356 ? Sl May14 9852:39 \_
/usr/lib/erlang/erts-5.9.1/bin/beam.smp -W w -K true -A30 -P 1048576 --
-root /usr/lib/erlang -progname erl -- -
rabbitmq 26605 0.0 0.0 10776 0 ? Ss May14 0:00 \_
inet_gethost 4
rabbitmq 26606 0.0 0.0 12876 36 ? S May14
0:00 \_ inet_gethost 4
As you can see, beam.smp is using 184% CPU and 82.4% of memory on this
server (4 cores, 1GB RAM).
Unfortunately, by the time I was able to ssh into the server to run
rabbitmqctl report, it said nodedown.
Here's the output of report after I manually killed beam.smp and ran
/etc/init.d/rabbitmq start:
Status of node 'rabbit at bt-ci-rabbitmq' ...
[{pid,18399},
{running_applications,[{rabbit,"RabbitMQ","2.8.2"},
{os_mon,"CPO CXC 138 46","2.2.9"},
{sasl,"SASL CXC 138 11","2.2.1"},
{mnesia,"MNESIA CXC 138 12","4.7"},
{stdlib,"ERTS CXC 138 10","1.18.1"},
{kernel,"ERTS CXC 138 10","2.15.1"}]},
{os,{unix,linux}},
{erlang_version,"Erlang R15B01 (erts-5.9.1) [source] [64-bit]
[smp:4:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,[{total,27528008},
{processes,10447908},
{processes_used,10447894},
{system,17080100},
{atom,504409},
{atom_used,475802},
{binary,806272},
{code,11905478},
{ets,823624}]},
{vm_memory_high_watermark,0.19999999923056544},
{vm_memory_limit,207944908},
{disk_free_limit,1039724544},
{disk_free,38120873984},
{file_descriptors,[{total_limit,924},
{total_used,6},
{sockets_limit,829},
{sockets_used,1}]},
{processes,[{limit,1048576},{used,130}]},
{run_queue,0},
{uptime,27}]
Cluster status of node 'rabbit at bt-ci-rabbitmq' ...
[{nodes,[{disc,['rabbit at bt-ci-rabbitmq']}]},
{running_nodes,['rabbit at bt-ci-rabbitmq']}]
Application environment of node 'rabbit at bt-ci-rabbitmq' ...
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue},
{cluster_nodes,[]},
{collect_statistics,none},
{collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,{mem_relative,1.0}},
{error_logger,{file,"/var/log/rabbitmq/rabbit at bt-ci-rabbitmq.log"}},
{frame_max,131072},
{hipe_compile,false},
{included_applications,[]},
{log_levels,[{connection,info}]},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{queue_index_max_journal_entries,262144},
{sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at bt-ci-rabbitmq-sasl.log"}},
{server_properties,[]},
{ssl_cert_login_from,distinguished_name},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,[binary,
{packet,raw},
{reuseaddr,true},
{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[5672]},
{trace_vhosts,[]},
{vm_memory_high_watermark,0.2}]
Connections:
Channels:
Queues on /:
pid name durable auto_delete arguments owner_pid
slave_pids synchronised_slave_pids exclusive_consumer_pid
exclusive_consumer_tag messages_ready messages_unacknowledged
messages consumers memory backing_queue_status
<'rabbit at bt-ci-rabbitmq'.2.186.0>
webhook_worker_master-testing_notifications true false
[{"x-ha-policy","all"}] [] [] 0 0 0 0
34688 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0},
{q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity},
{ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,0},
{persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0},
{avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}, {mirror_seen,0},
{mirror_senders,0}]
<'rabbit at bt-ci-rabbitmq'.2.187.0> gateway_master_test true
false [{"x-ha-policy","all"}] [] [] 79 0
79 0 88808 [{q1,0}, {q2,0},
{delta,{delta,undefined,0,undefined}}, {q3,79}, {q4,0}, {len,79},
{pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0},
{ram_ack_count,0}, {next_seq_id,16384}, {persistent_count,79},
{avg_ingress_rate,0.0}, {avg_egress_rate,0.0},
{avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}, {mirror_seen,0},
{mirror_senders,0}]
<'rabbit at bt-ci-rabbitmq'.2.188.0> gateway_release_test true
false [{"x-ha-policy","all"}] [] [] 109 0
109 0 142928 [{q1,0}, {q2,0},
{delta,{delta,undefined,0,undefined}}, {q3,109}, {q4,0}, {len,109},
{pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0},
{ram_ack_count,0}, {next_seq_id,16384}, {persistent_count,109},
{avg_ingress_rate,0.0}, {avg_egress_rate,0.0},
{avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}, {mirror_seen,0},
{mirror_senders,0}]
Exchanges on /:
name type durable auto_delete internal arguments
amq.direct direct true false false []
amq.topic topic true false false []
amq.rabbitmq.trace topic true false false []
amq.rabbitmq.log topic true false false []
amq.fanout fanout true false false []
amq.headers headers true false false []
direct true false false []
amq.match headers true false false []
Bindings on /:
source_name source_kind destination_name destination_kind
routing_key arguments
exchange gateway_master_test queue gateway_master_test []
exchange gateway_release_test queue gateway_release_test []
exchange webhook_worker_master-testing_notifications queue
webhook_worker_master-testing_notifications []
Consumers on /:
Permissions on /:
user configure write read
guest .* .* .*
End of server status report
...done.
More information about the rabbitmq-discuss
mailing list