<div dir="ltr">The UI is not accessible , The 3-nodes are clustered with HA two policy<div><br></div><div>rabbitmqctl reports shows </div><div><br></div><div><div>Reporting server status on {{2014,8,26},{18,39,31}}</div><div>
<br></div><div> ...</div><div>Status of node rabbit@adsmv0020 ...</div><div>[{pid,22153},</div><div> {running_applications,</div><div> [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},</div>
<div> {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},</div><div> {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"},</div><div> {rabbit,"RabbitMQ","3.2.4"},</div>
<div> {os_mon,"CPO CXC 138 46","2.2.7"},</div><div> {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"},</div><div> {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},</div>
<div> {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},</div><div> {xmerl,"XML parser","1.2.10"},</div><div> {inets,"INETS CXC 138 49","5.7.1"},</div>
<div> {mnesia,"MNESIA CXC 138 12","4.5"},</div><div> {amqp_client,"RabbitMQ AMQP Client","3.2.4"},</div><div> {sasl,"SASL CXC 138 11","2.1.10"},</div>
<div> {stdlib,"ERTS CXC 138 10","1.17.5"},</div><div> {kernel,"ERTS CXC 138 10","2.14.5"}]},</div><div> {os,{unix,linux}},</div><div> {erlang_version,</div><div> "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},</div>
<div> {memory,</div><div> [{total,2285543720},</div><div> {connection_procs,207275088},</div><div> {queue_procs,599282104},</div><div> {plugins,234739776},</div><div> {other_proc,309911344},</div><div>
{mnesia,289075584},</div><div> {mgmt_db,9096},</div><div> {msg_index,14688048},</div><div> {other_ets,28236624},</div><div> {binary,290512400},</div><div> {code,17776116},</div><div> {atom,1621321},</div>
<div> {other_system,292416219}]},</div><div> {vm_memory_high_watermark,0.7},</div><div> {vm_memory_limit,11753753804},</div><div> {disk_free_limit,1000000000},</div><div> {disk_free,19386892288},</div><div> {file_descriptors,</div>
<div> [{total_limit,327579},</div><div> {total_used,3137},</div><div> {sockets_limit,294819},</div><div> {sockets_used,3135}]},</div><div> {processes,[{limit,1048576},{used,144321}]},</div><div> {run_queue,0},</div>
<div> {uptime,6267}]</div></div><div><br></div><div><br></div><div><div>Cluster status of node rabbit@adsmv0020 ...</div><div>[{nodes,[{disc,[rabbit@adsmv0019,rabbit@adsmv0018]},{ram,[rabbit@adsmv0020]}]},</div><div> {running_nodes,[rabbit@adsmv0018,rabbit@adsmv0019,rabbit@adsmv0020]},</div>
<div> {partitions,[]}]</div><div><br></div><div>Application environment of node rabbit@adsmv0020 ...</div><div>[{auth_backends,[rabbit_auth_backend_internal]},</div><div> {auth_mechanisms,['PLAIN','AMQPLAIN']},</div>
<div> {backing_queue_module,rabbit_variable_queue},</div><div> {cluster_nodes,{[],disc}},</div><div> {cluster_partition_handling,autoheal},</div><div> {collect_statistics,fine},</div><div> {collect_statistics_interval,5000},</div>
<div> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},</div><div> {default_user,<<"guest">>},</div><div> {default_user_tags,[administrator]},</div>
<div> {default_vhost,<<"/">>},</div><div> {delegate_count,16},</div><div> {disk_free_limit,1000000000},</div><div> {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},</div><div> {error_logger,{file,"/var/log/rabbitmq/rabbit@adsmv0020.log"}},</div>
<div> {frame_max,131072},</div><div> {halt_on_upgrade_failure,true},</div><div> {heartbeat,580},</div><div> {hipe_compile,false},</div><div> {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,</div><div>
rabbit_command_assembler,rabbit_framing_amqp_0_9_1,</div><div> rabbit_basic,rabbit_event,lists,queue,priority_queue,</div><div> rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,</div>
<div> rabbit_exchange_type_direct,rabbit_guid,rabbit_net,</div><div> rabbit_amqqueue_process,rabbit_variable_queue,</div><div> rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,</div>
<div> sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,</div><div> rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,</div><div> file_handle_cache,rabbit_msg_store,array,</div>
<div> rabbit_msg_store_ets_index,rabbit_msg_file,</div><div> rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,</div><div> mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,</div>
<div> ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,</div><div> ssl]},</div><div> {included_applications,[]},</div><div> {log_levels,[{connection,info}]},</div><div> {msg_store_file_size_limit,16777216},</div>
<div> {msg_store_index_module,rabbit_msg_store_ets_index},</div><div> {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},</div><div> {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0020-plugins-expand"},</div>
<div> {queue_index_max_journal_entries,65536},</div><div> {reverse_dns_lookups,true},</div><div> {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit@adsmv0020-sasl.log"}},</div></div><div><div> {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit@adsmv0020-sasl.log"}},</div>
<div> {server_properties,[]},</div><div> {ssl_apps,[asn1,crypto,public_key,ssl]},</div><div> {ssl_cert_login_from,distinguished_name},</div><div> {ssl_listeners,[]},</div><div> {ssl_options,[]},</div><div> {tcp_listen_options,[binary,</div>
<div> {packet,raw},</div><div> {reuseaddr,true},</div><div> {backlog,128},</div><div> {nodelay,true},</div><div> {linger,{true,0}},</div>
<div> {exit_on_close,false}]},</div><div> {tcp_listeners,[5672]},</div><div> {trace_vhosts,[]},</div><div> {vm_memory_high_watermark,0.7},</div><div> {vm_memory_high_watermark_paging_ratio,0.7}]</div>
<div><br></div><div>Status of node rabbit@adsmv0019 ...</div><div>[{pid,27139},</div><div> {running_applications,</div><div> [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},</div><div> {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},</div>
<div> {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"},</div><div> {rabbit,"RabbitMQ","3.2.4"},</div><div> {os_mon,"CPO CXC 138 46","2.2.7"},</div>
<div> {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"},</div><div> {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},</div><div> {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},</div>
<div> {xmerl,"XML parser","1.2.10"},</div><div> {inets,"INETS CXC 138 49","5.7.1"},</div><div> {mnesia,"MNESIA CXC 138 12","4.5"},</div><div> {amqp_client,"RabbitMQ AMQP Client","3.2.4"},</div>
<div> {sasl,"SASL CXC 138 11","2.1.10"},</div><div> {stdlib,"ERTS CXC 138 10","1.17.5"},</div><div> {kernel,"ERTS CXC 138 10","2.14.5"}]},</div>
<div> {os,{unix,linux}},</div><div> {erlang_version,</div><div> "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},</div><div> {memory,</div><div> [{total,2375638736},</div>
<div> {connection_procs,228104952},</div><div> {queue_procs,643238504},</div><div> {plugins,259036872},</div><div> {other_proc,276878920},</div><div> {mnesia,288547136},</div><div> {mgmt_db,9096},</div>
<div> {msg_index,14503600},</div><div> {other_ets,30000664},</div><div> {binary,321298960},</div></div><div><div> {binary,321298960},</div><div> {code,17776116},</div><div> {atom,1621321},</div>
<div> {other_system,294622595}]},</div><div> {vm_memory_high_watermark,0.7},</div><div> {vm_memory_limit,11753753804},</div><div> {disk_free_limit,1000000000},</div><div> {disk_free,19439706112},</div><div> {file_descriptors,</div>
<div> [{total_limit,327579},</div><div> {total_used,3466},</div><div> {sockets_limit,294819},</div><div> {sockets_used,3464}]},</div><div> {processes,[{limit,1048576},{used,159459}]},</div><div> {run_queue,0},</div>
<div> {uptime,6302}]</div><div><br></div><div>Cluster status of node rabbit@adsmv0019 ...</div><div>[{nodes,[{disc,[rabbit@adsmv0018,rabbit@adsmv0019]},{ram,[rabbit@adsmv0020]}]},</div><div> {running_nodes,[rabbit@adsmv0020,rabbit@adsmv0018,rabbit@adsmv0019]},</div>
<div> {partitions,[]}]</div><div><br></div><div>Application environment of node rabbit@adsmv0019 ...</div><div>[{auth_backends,[rabbit_auth_backend_internal]},</div><div> {auth_mechanisms,['PLAIN','AMQPLAIN']},</div>
<div> {backing_queue_module,rabbit_variable_queue},</div><div> {cluster_nodes,{[],disc}},</div><div> {cluster_partition_handling,autoheal},</div><div> {collect_statistics,fine},</div><div> {collect_statistics_interval,5000},</div>
<div> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},</div><div> {default_user,<<"guest">>},</div><div> {default_user_tags,[administrator]},</div>
<div> {default_vhost,<<"/">>},</div><div> {delegate_count,16},</div><div> {disk_free_limit,1000000000},</div><div> {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},</div><div> {error_logger,{file,"/var/log/rabbitmq/rabbit@adsmv0019.log"}},</div>
<div> {frame_max,131072},</div><div> {halt_on_upgrade_failure,true},</div><div> {heartbeat,580},</div><div> {hipe_compile,false},</div><div> {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,</div><div>
rabbit_command_assembler,rabbit_framing_amqp_0_9_1,</div></div><div><div> rabbit_command_assembler,rabbit_framing_amqp_0_9_1,</div><div> rabbit_basic,rabbit_event,lists,queue,priority_queue,</div>
<div> rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,</div><div> rabbit_exchange_type_direct,rabbit_guid,rabbit_net,</div><div> rabbit_amqqueue_process,rabbit_variable_queue,</div>
<div> rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,</div><div> sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,</div><div> rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,</div>
<div> file_handle_cache,rabbit_msg_store,array,</div><div> rabbit_msg_store_ets_index,rabbit_msg_file,</div><div> rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,</div>
<div> mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,</div><div> ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,</div><div> ssl]},</div><div> {included_applications,[]},</div>
<div> {log_levels,[{connection,info}]},</div><div> {msg_store_file_size_limit,16777216},</div><div> {msg_store_index_module,rabbit_msg_store_ets_index},</div><div> {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},</div>
<div> {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0019-plugins-expand"},</div><div> {queue_index_max_journal_entries,65536},</div><div> {reverse_dns_lookups,true},</div><div> {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit@adsmv0019-sasl.log"}},</div>
<div> {server_properties,[]},</div><div> {ssl_apps,[asn1,crypto,public_key,ssl]},</div><div> {ssl_cert_login_from,distinguished_name},</div><div> {ssl_listeners,[]},</div><div> {ssl_options,[]},</div><div> {tcp_listen_options,[binary,</div>
<div> {packet,raw},</div><div> {reuseaddr,true},</div><div> {backlog,128},</div><div> {nodelay,true},</div><div> {linger,{true,0}},</div>
<div> {exit_on_close,false}]},</div><div> {tcp_listeners,[5672]},</div><div> {trace_vhosts,[]},</div><div> {vm_memory_high_watermark,0.7},</div><div> {vm_memory_high_watermark_paging_ratio,0.7}]</div>
<div><br></div><div>Status of node rabbit@adsmv0018 ...</div><div>[{pid,20727},</div><div> {running_applications,</div><div> [{rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.2.4"},</div><div> {rabbitmq_management,"RabbitMQ Management Console","3.2.4"},</div>
<div> {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.4"},</div><div> {rabbit,"RabbitMQ","3.2.4"},</div><div><div> {os_mon,"CPO CXC 138 46","2.2.7"},</div>
<div> {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.4"},</div><div> {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"},</div><div> {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.4-git680dba8"},</div>
<div> {xmerl,"XML parser","1.2.10"},</div><div> {inets,"INETS CXC 138 49","5.7.1"},</div><div> {mnesia,"MNESIA CXC 138 12","4.5"},</div><div> {amqp_client,"RabbitMQ AMQP Client","3.2.4"},</div>
<div> {sasl,"SASL CXC 138 11","2.1.10"},</div><div> {stdlib,"ERTS CXC 138 10","1.17.5"},</div><div> {kernel,"ERTS CXC 138 10","2.14.5"}]},</div>
<div> {os,{unix,linux}},</div><div> {erlang_version,</div><div> "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [kernel-poll:true]\n"},</div><div> {memory,</div><div> [{total,6095435856},</div>
<div> {connection_procs,185360136},</div><div> {queue_procs,526782896},</div><div> {plugins,-1872186488},</div><div> {other_proc,2447546120},</div><div> {mnesia,289695904},</div><div> {mgmt_db,3870226464},</div>
<div> {msg_index,23253920},</div><div> {other_ets,51366128},</div><div> {binary,249989160},</div><div> {code,17776116},</div><div> {atom,1622129},</div><div> {other_system,304003371}]},</div>
<div> {vm_memory_high_watermark,0.7},</div><div> {vm_memory_limit,11753753804},</div><div> {disk_free_limit,1000000000},</div><div> {disk_free,19338166272},</div><div> {file_descriptors,</div><div> [{total_limit,327579},</div>
<div> {total_used,2748},</div><div> {sockets_limit,294819},</div><div> {sockets_used,2746}]},</div><div> {processes,[{limit,1048576},{used,126430}]},</div><div> {run_queue,0},</div><div> {uptime,6789}]</div>
<div><br></div><div>Cluster status of node rabbit@adsmv0018 ...</div><div>[{nodes,[{disc,[rabbit@adsmv0018,rabbit@adsmv0019]},{ram,[rabbit@adsmv0020]}]},</div><div> {running_nodes,[rabbit@adsmv0020,rabbit@adsmv0019,rabbit@adsmv0018]},</div>
<div> {partitions,[]}]</div><div><br></div><div>Application environment of node rabbit@adsmv0018 ...</div><div>[{auth_backends,[rabbit_auth_backend_internal]},</div><div> {auth_mechanisms,['PLAIN','AMQPLAIN']},</div>
<div> {backing_queue_module,rabbit_variable_queue},</div><div> {cluster_nodes,{[],disc}},</div></div></div><div><div> {cluster_partition_handling,autoheal},</div><div> {collect_statistics,fine},</div><div> {collect_statistics_interval,5000},</div>
<div> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},</div><div> {default_user,<<"guest">>},</div><div> {default_user_tags,[administrator]},</div>
<div> {default_vhost,<<"/">>},</div><div> {delegate_count,16},</div><div> {disk_free_limit,1000000000},</div><div> {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},</div><div> {error_logger,{file,"/var/log/rabbitmq/rabbit@adsmv0018.log"}},</div>
<div> {frame_max,131072},</div><div> {halt_on_upgrade_failure,true},</div><div> {heartbeat,580},</div><div> {hipe_compile,false},</div><div> {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,</div><div>
rabbit_command_assembler,rabbit_framing_amqp_0_9_1,</div><div> rabbit_basic,rabbit_event,lists,queue,priority_queue,</div><div> rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,</div>
<div> rabbit_exchange_type_direct,rabbit_guid,rabbit_net,</div><div> rabbit_amqqueue_process,rabbit_variable_queue,</div><div> rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,</div>
<div> sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,</div><div> rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,</div><div> file_handle_cache,rabbit_msg_store,array,</div>
<div> rabbit_msg_store_ets_index,rabbit_msg_file,</div><div> rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,</div><div> mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,</div>
<div> ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,</div><div> ssl]},</div><div> {included_applications,[]},</div><div> {log_levels,[{connection,info}]},</div><div> {msg_store_file_size_limit,16777216},</div>
<div> {msg_store_index_module,rabbit_msg_store_ets_index},</div><div> {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins"},</div><div> {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@adsmv0018-plugins-expand"},</div>
<div> {queue_index_max_journal_entries,65536},</div><div> {reverse_dns_lookups,true},</div><div> {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit@adsmv0018-sasl.log"}},</div><div> {server_properties,[]},</div>
<div> {ssl_apps,[asn1,crypto,public_key,ssl]},</div><div> {ssl_cert_login_from,distinguished_name},</div><div> {ssl_listeners,[]},</div><div> {ssl_options,[]},</div><div> {tcp_listen_options,[binary,</div><div> {packet,raw},</div>
<div> {reuseaddr,true},</div></div><div><div> {backlog,128},</div><div> {nodelay,true},</div><div> {linger,{true,0}},</div><div> {exit_on_close,false}]},</div>
<div> {tcp_listeners,[5672]},</div><div> {trace_vhosts,[]},</div><div> {vm_memory_high_watermark,0.7},</div><div> {vm_memory_high_watermark_paging_ratio,0.7}]</div><div><br></div></div><div><br></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Tue, Aug 26, 2014 at 11:42 AM, Michael Klishin <span dir="ltr"><<a href="mailto:mklishin@pivotal.io" target="_blank">mklishin@pivotal.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="">On 26 August 2014 at 20:02:09, srikanth tns (<a href="mailto:srikanthtns@gmail.com">srikanthtns@gmail.com</a>) wrote:<br>
</div><div class="">> > netstat -tan | grep ':15672' | awk '{print $6 }' | sort | uniq -c<br>
> 3 CLOSE_WAIT<br>
> 1882 ESTABLISHED<br>
> 1 LISTEN<br>
> 10 TIME_WAIT<br>
<br>
</div>That's about 1900 concurrent connections to a single node. Without knowing<br>
how many channels can be open on the open ones, that's still a fairly<br>
large number for a single node.<br>
<br>
Consider clustering this node with a couple more and spread the connections<br>
between them, e.g. using an HAproxy instance in front of the cluster. <br>
<div class="HOEnZb"><div class="h5">--<br>
MK<br>
<br>
Staff Software Engineer, Pivotal/RabbitMQ<br>
</div></div></blockquote></div><br></div>