[rabbitmq-discuss] RabbitMQ fails to close connections
Tuure Laurinolli
tuure.laurinolli at portalify.com
Mon Jul 29 14:03:32 BST 2013
Hi,
When I uncleanly disconnect a client, it is retained on RabbitMQ side apparently forever. I would not be very surprised about this if it weren't for the fact that the connection has heartbeats enabled. I would expect the server to attempt to ping the client and to disconnect it if no response is received.
The offending connections is last on the report below. As can be seen from netstat output, the corresponding TCP connection no longer exists.
Report output:
Reporting server status on {{2013,7,29},{13,0,53}}
...
Status of node 'rabbit at pmc-localtest' ...
[{pid,8352},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.1.1"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.1.1"},
{webmachine,"webmachine","1.9.1-rmq3.1.1-git52e62bc"},
{mochiweb,"MochiMedia Web Server","2.3.1-rmq3.1.1-gitd541e9a"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.1"},
{rabbit,"RabbitMQ","3.1.1"},
{os_mon,"CPO CXC 138 46","2.2.9"},
{inets,"INETS CXC 138 49","5.9"},
{xmerl,"XML parser","1.3.1"},
{mnesia,"MNESIA CXC 138 12","4.7"},
{amqp_client,"RabbitMQ AMQP Client","3.1.1"},
{sasl,"SASL CXC 138 11","2.2.1"},
{stdlib,"ERTS CXC 138 10","1.18.1"},
{kernel,"ERTS CXC 138 10","2.15.1"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,
[{total,33681696},
{connection_procs,170720},
{queue_procs,178216},
{plugins,434424},
{other_proc,9304848},
{mnesia,83904},
{mgmt_db,127024},
{msg_index,43776},
{other_ets,1102232},
{binary,97744},
{code,18464238},
{atom,703377},
{other_system,2971193}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,206723481},
{disk_free_limit,1000000000},
{disk_free,882683904},
{file_descriptors,
[{total_limit,924},{total_used,13},{sockets_limit,829},{sockets_used,4}]},
{processes,[{limit,1048576},{used,224}]},
{run_queue,0},
{uptime,3300}]
Cluster status of node 'rabbit at pmc-localtest' ...
[{nodes,[{disc,['rabbit at pmc-localtest']}]},
{running_nodes,['rabbit at pmc-localtest']},
{partitions,[]}]
Application environment of node 'rabbit at pmc-localtest' ...
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{backing_queue_module,rabbit_variable_queue},
{cluster_nodes,{[],disc}},
{cluster_partition_handling,ignore},
{collect_statistics,fine},
{collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,1000000000},
{enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
{error_logger,{file,"/var/log/rabbitmq/rabbit at pmc-localtest.log"}},
{frame_max,131072},
{heartbeat,600},
{hipe_compile,false},
{included_applications,[]},
{log_levels,[{connection,info}]},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.1/sbin/../plugins"},
{plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit at pmc-localtest-plugins-expand"},
{queue_index_max_journal_entries,65536},
{reverse_dns_lookups,false},
{sasl_error_logger,{file,"/var/log/rabbitmq/rabbit at pmc-localtest-sasl.log"}},
{server_properties,[]},
{ssl_cert_login_from,distinguished_name},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,[binary,
{packet,raw},
{reuseaddr,true},
{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[5672]},
{trace_vhosts,[]},
{vm_memory_high_watermark,0.4}]
Connections:
pid name port peer_port host peer_host ssl peer_cert_subject peer_cert_issuer peer_cert_validity auth_mechanism ssl_protocol ssl_key_exchange ssl_cipher ssl_hash protocol user vhost timeout frame_max client_properties recv_oct recv_cnt send_oct send_cnt send_pend state last_blocked_by last_blocked_agechannels
<'rabbit at pmc-localtest'.3.290.0> 127.0.0.1:45757 -> 127.0.0.1:5672 5672 45757 127.0.0.1 127.0.0.1 false PLAIN {0,9,1} guest / 0 131072 [{"product","rabbitmq-c"},{"information","See http://hg.rabbitmq.com/rabbitmq-c/"}] 370 9 492 8 0 blocking none infinity 1
<'rabbit at pmc-localtest'.3.298.0> 127.0.0.1:45758 -> 127.0.0.1:5672 5672 45758 127.0.0.1 127.0.0.1 false PLAIN {0,9,1} guest / 10 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2012 VMware, Inc."},{"version","2.8.4"}] 6088 671 3244 341 0 blocking none infinity 1
<'rabbit at pmc-localtest'.3.318.0> 127.0.0.1:45760 -> 127.0.0.1:5672 5672 45760 127.0.0.1 127.0.0.1 false PLAIN {0,9,1} guest / 5 131072 [{"product","rabbitmq-c"},{"information","See http://hg.rabbitmq.com/rabbitmq-c/"}] 728 9 993 76 0 blocked resource 3295.501924 1
Channels:
pid name connection number user vhost transactional confirm consumer_count messages_unacknowledged messages_unconfirmed messages_uncommitted acks_uncommitted prefetch_count client_flow_blocked
<'rabbit at pmc-localtest'.3.295.0> 127.0.0.1:45757 -> 127.0.0.1:5672 (1) <'rabbit at pmc-localtest'.3.290.0> 1 guest / false false 1 0 0 0 0 0 false
<'rabbit at pmc-localtest'.3.305.0> 127.0.0.1:45758 -> 127.0.0.1:5672 (1) <'rabbit at pmc-localtest'.3.298.0> 1 guest / false false 1 0 0 0 0 1 false
<'rabbit at pmc-localtest'.3.325.0> 127.0.0.1:45760 -> 127.0.0.1:5672 (1) <'rabbit at pmc-localtest'.3.318.0> 1 guest / false false 0 0 0 0 0 0 false
Queues on /:
pid name durable auto_delete arguments owner_pid policy exclusive_consumer_pid exclusive_consumer_tag messages_ready messages_unacknowledged messages consumers memory slave_pids synchronised_slave_pids backing_queue_status status
<'rabbit at pmc-localtest'.3.207.0> queue.pmc.smpp-in true false [] 0 0 0 0 21800 [{q1,0}, {q2,0}, {delta,{delta,0,0,0}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}] running
<'rabbit at pmc-localtest'.3.208.0> queue.pmc.email-in true false [] 0 0 0 0 21800 [{q1,0}, {q2,0}, {delta,{delta,0,0,0}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}] running
<'rabbit at pmc-localtest'.3.209.0> queue.pmc.router-in true false [] 0 0 0 1 21944 [{q1,0}, {q2,0}, {delta,{delta,0,0,0}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}] running
<'rabbit at pmc-localtest'.3.210.0> queue.pmc.cassidian-in true false [] 0 0 0 0 21800 [{q1,0}, {q2,0}, {delta,{delta,0,0,0}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}] running
<'rabbit at pmc-localtest'.3.211.0> queue.pmc.sf-in true false [] 0 0 0 0 21800 [{q1,0}, {q2,0}, {delta,{delta,0,0,0}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}] running
<'rabbit at pmc-localtest'.3.212.0> queue.pmc.astrid-112-in true false [] 0 0 0 1 22016 [{q1,0}, {q2,0}, {delta,{delta,0,0,0}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}] running
<'rabbit at pmc-localtest'.3.213.0> queue.pmc.tetra-pei-in true false [] 0 0 0 0 21800 [{q1,0}, {q2,0}, {delta,{delta,0,0,0}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}] running
<'rabbit at pmc-localtest'.3.326.0> queue.test.sf-in false false [] <'rabbit at pmc-localtest'.3.318.0> 0 0 0 0 14008 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}] running
Exchanges on /:
name type durable auto_delete internal arguments policy
direct true false false []
amq.direct direct true false false []
amq.fanout fanout true false false []
amq.headers headers true false false []
amq.match headers true false false []
amq.rabbitmq.log topic true false false []
amq.rabbitmq.trace topic true false false []
amq.topic topic true false false []
exchange.pmc.astrid-112-in direct true false false []
exchange.pmc.cassidian-in direct true false false []
exchange.pmc.email-in fanout true false false []
exchange.pmc.router-in fanout true false false []
exchange.pmc.sf-in direct true false false []
exchange.pmc.smpp-in fanout true false false []
Bindings on /:
source_name source_kind destination_name destination_kind routing_key arguments
exchange queue.pmc.astrid-112-in queue queue.pmc.astrid-112-in []
exchange queue.pmc.cassidian-in queue queue.pmc.cassidian-in []
exchange queue.pmc.email-in queue queue.pmc.email-in []
exchange queue.pmc.router-in queue queue.pmc.router-in []
exchange queue.pmc.sf-in queue queue.pmc.sf-in []
exchange queue.pmc.smpp-in queue queue.pmc.smpp-in []
exchange queue.pmc.tetra-pei-in queue queue.pmc.tetra-pei-in []
exchange queue.test.sf-in queue queue.test.sf-in []
exchange.pmc.astrid-112-in exchange queue.pmc.astrid-112-in queue []
exchange.pmc.cassidian-in exchange queue.pmc.cassidian-in queue []
exchange.pmc.email-in exchange queue.pmc.email-in queue []
exchange.pmc.router-in exchange queue.pmc.router-in queue []
exchange.pmc.sf-in exchange queue.pmc.sf-in queue []
exchange.pmc.sf-in exchange queue.test.sf-in queue []
exchange.pmc.smpp-in exchange queue.pmc.smpp-in queue []
Consumers on /:
queue_name channel_pid consumer_tag ack_required
queue.pmc.astrid-112-in <'rabbit at pmc-localtest'.3.305.0> amq.ctag-xl4tPpQSCoOOueAao0br6g true
queue.pmc.router-in <'rabbit at pmc-localtest'.3.295.0> amq.ctag-oeR9Nin_ofEY7vvQzfWm-g false
Permissions on /:
user configure write read
admin .* .* .*
guest .* .* .*
Policies on /:
Parameters on /:
...done.
Netstat output:
pmc-localtest:~ # netstat -np | grep 5672
tcp 0 0 127.0.0.1:45757 127.0.0.1:5672 ESTABLISHED 22394/perl
tcp 0 0 127.0.0.1:5672 127.0.0.1:45757 ESTABLISHED 8352/beam
tcp 0 0 127.0.0.1:5672 127.0.0.1:45758 ESTABLISHED 8352/beam
tcp 0 0 127.0.0.1:45758 127.0.0.1:5672 ESTABLISHED 3530/java
More information about the rabbitmq-discuss
mailing list