I have been testing my cluster and I have come across an unexpected behavior. As explained in the subject, the cluster runs well on 3 nodes. If I bring 1 node down, it still runs smoothly. But if I bring a second node down, the third one becomes unresponsive.<div>This is what the third node provides after running "rabbitmqctl report":</div><div><br></div><div>
<blockquote>Reporting server status on {{2013,8,6},{15,19,8}}<br> ...<br>Error: {aborted,{no_exists,rabbit_vhost}}</blockquote><div><br></div><div>Then if I bring 1 or 2 of my nodes back up, the third one becomes responsive and everything is back to normal. </div><div><br></div><div>Here is the report after bringing the nodes up again:</div><div><br></div><blockquote>Reporting server status on {{2013,8,6},{15,26,31}}<br> ...<br>Status of node rabbit@qcluster1 ...<br>[{pid,901},<br> {running_applications,<br> [{rabbitmq_management,"RabbitMQ Management Console","3.1.3"},<br> {rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.3"},<br> {rabbit,"RabbitMQ","3.1.3"},<br> {os_mon,"CPO CXC 138 46","2.2.7"},<br> {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.1.3"},<br> {webmachine,"webmachine","1.9.1-rmq3.1.3-git52e62bc"},<br> {mochiweb,"MochiMedia Web Server","2.3.1-rmq3.1.3-gitd541e9a"},<br> {xmerl,"XML parser","1.2.10"},<br> {inets,"INETS CXC 138 49","5.7.1"},<br> {mnesia,"MNESIA CXC 138 12","4.5"},<br> {amqp_client,"RabbitMQ AMQP Client","3.1.3"},<br> {sasl,"SASL CXC 138 11","2.1.10"},<br> {stdlib,"ERTS CXC 138 10","1.17.5"},<br> {kernel,"ERTS CXC 138 10","2.14.5"}]},<br> {os,{unix,linux}},<br> {erlang_version,<br> "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},<br> {memory,<br> [{total,31689208},<br> {connection_procs,5408},<br> {queue_procs,5408},<br> {plugins,340592},<br> {other_proc,9166952},<br> {mnesia,61984},<br> {mgmt_db,10248},<br> {msg_index,34160},<br> {other_ets,1080168},<br> {binary,6480},<br> {code,17557932},<br> {atom,1565809},<br> {other_system,1854067}]},<br> {vm_memory_high_watermark,0.4},<br> {vm_memory_limit,153295257},<br> {disk_free_limit,1000000000},<br> {disk_free,77554466816},<br> {file_descriptors,<br> [{total_limit,924},{total_used,3},{sockets_limit,829},{sockets_used,1}]},<br> {processes,[{limit,1048576},{used,187}]},<br> {run_queue,0},<br> {uptime,210}]<br>Cluster status of node rabbit@qcluster1 ...<br>[{nodes,[{disc,[rabbit@qcluster2,rabbit@qcluster0]},{ram,[rabbit@qcluster1]}]},<br> {running_nodes,[rabbit@qcluster0,rabbit@qcluster2,rabbit@qcluster1]},<br> {partitions,[]}]<br>Application environment of node rabbit@qcluster1 ...<br>[{auth_backends,[rabbit_auth_backend_internal]},<br> {auth_mechanisms,['PLAIN','AMQPLAIN']},<br> {backing_queue_module,rabbit_variable_queue},<br> {cluster_nodes,{[rabbit@qcluster0,rabbit@qcluster1],ram}},<br> {cluster_partition_handling,pause_minority},<br> {collect_statistics,fine},<br> {collect_statistics_interval,5000},<br> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},<br> {default_user,<<"guest">>},<br> {default_user_tags,[administrator]},<br> {default_vhost,<<"/">>},<br> {delegate_count,16},<br> {disk_free_limit,1000000000},<br> {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},<br> {error_logger,{file,"/var/log/rabbitmq/rabbit@qcluster1.log"}},<br> {frame_max,131072},<br> {heartbeat,600},<br> {hipe_compile,false},<br> {included_applications,[]},<br> {log_levels,[{connection,info}]},<br> {msg_store_file_size_limit,16777216},<br> {msg_store_index_module,rabbit_msg_store_ets_index},<br> {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.3/sbin/../plugins"},<br> {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@qcluster1-plugins-expand"},<br> {queue_index_max_journal_entries,65536},<br> {reverse_dns_lookups,false},<br> {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit@qcluster1-sasl.log"}},<br> {server_properties,[]},<br> {ssl_cert_login_from,distinguished_name},<br> {ssl_listeners,[]},<br> {ssl_options,[]},<br> {tcp_listen_options,[binary,<br> {packet,raw},<br> {reuseaddr,true},<br> {backlog,128},<br> {nodelay,true},<br> {linger,{true,0}},<br> {exit_on_close,false}]},<br> {tcp_listeners,[{"auto",5672}]},<br> {trace_vhosts,[]},<br> {vm_memory_high_watermark,0.4}]<br>Status of node rabbit@qcluster2 ...<br>[{pid,1940},<br> {running_applications,<br> [{rabbitmq_management,"RabbitMQ Management Console","3.1.3"},<br> {rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.3"},<br> {rabbit,"RabbitMQ","3.1.3"},<br> {os_mon,"CPO CXC 138 46","2.2.7"},<br> {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.1.3"},<br> {webmachine,"webmachine","1.9.1-rmq3.1.3-git52e62bc"},<br> {mochiweb,"MochiMedia Web Server","2.3.1-rmq3.1.3-gitd541e9a"},<br> {mnesia,"MNESIA CXC 138 12","4.5"},<br> {amqp_client,"RabbitMQ AMQP Client","3.1.3"},<br> {xmerl,"XML parser","1.2.10"},<br> {inets,"INETS CXC 138 49","5.7.1"},<br> {sasl,"SASL CXC 138 11","2.1.10"},<br> {stdlib,"ERTS CXC 138 10","1.17.5"},<br> {kernel,"ERTS CXC 138 10","2.14.5"}]},<br> {os,{unix,linux}},<br> {erlang_version,<br> "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},<br> {memory,<br> [{total,32032800},<br> {connection_procs,5408},<br> {queue_procs,5408},<br> {plugins,315040},<br> {other_proc,9172832},<br> {mnesia,61952},<br> {mgmt_db,89704},<br> {msg_index,34160},<br> {other_ets,1111736},<br> {binary,33384},<br> {code,17743902},<br> {atom,1603433},<br> {other_system,1855841}]},<br> {vm_memory_high_watermark,0.4},<br> {vm_memory_limit,153295257},<br> {disk_free_limit,1000000000},<br> {disk_free,77553033216},<br> {file_descriptors,<br> [{total_limit,924},{total_used,3},{sockets_limit,829},{sockets_used,1}]},<br> {processes,[{limit,1048576},{used,191}]},<br> {run_queue,0},<br> {uptime,617}]<br>Cluster status of node rabbit@qcluster2 ...<br>[{nodes,[{disc,[rabbit@qcluster0,rabbit@qcluster2]},{ram,[rabbit@qcluster1]}]},<br> {running_nodes,[rabbit@qcluster1,rabbit@qcluster0,rabbit@qcluster2]},<br> {partitions,[]}]<br>Application environment of node rabbit@qcluster2 ...<br>[{auth_backends,[rabbit_auth_backend_internal]},<br> {auth_mechanisms,['PLAIN','AMQPLAIN']},<br> {backing_queue_module,rabbit_variable_queue},<br> {cluster_nodes,{[rabbit@qcluster0,rabbit@qcluster1,rabbit@qcluster2],ram}},<br> {cluster_partition_handling,pause_minority},<br> {collect_statistics,fine},<br> {collect_statistics_interval,5000},<br> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},<br> {default_user,<<"guest">>},<br> {default_user_tags,[administrator]},<br> {default_vhost,<<"/">>},<br> {delegate_count,16},<br> {disk_free_limit,1000000000},<br> {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},<br> {error_logger,{file,"/var/log/rabbitmq/rabbit@qcluster2.log"}},<br> {frame_max,131072},<br> {heartbeat,600},<br> {hipe_compile,false},<br> {included_applications,[]},<br> {log_levels,[{connection,info}]},<br> {msg_store_file_size_limit,16777216},<br> {msg_store_index_module,rabbit_msg_store_ets_index},<br> {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.3/sbin/../plugins"},<br> {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@qcluster2-plugins-expand"},<br> {queue_index_max_journal_entries,65536},<br> {reverse_dns_lookups,false},<br> {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit@qcluster2-sasl.log"}},<br> {server_properties,[]},<br> {ssl_cert_login_from,distinguished_name},<br> {ssl_listeners,[]},<br> {ssl_options,[]},<br> {tcp_listen_options,[binary,<br> {packet,raw},<br> {reuseaddr,true},<br> {backlog,128},<br> {nodelay,true},<br> {linger,{true,0}},<br> {exit_on_close,false}]},<br> {tcp_listeners,[{"auto",5672}]},<br> {trace_vhosts,[]},<br> {vm_memory_high_watermark,0.4}]<br>Status of node rabbit@qcluster0 ...<br>[{pid,873},<br> {running_applications,<br> [{rabbitmq_management,"RabbitMQ Management Console","3.1.3"},<br> {rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.3"},<br> {rabbit,"RabbitMQ","3.1.3"},<br> {os_mon,"CPO CXC 138 46","2.2.7"},<br> {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.1.3"},<br> {webmachine,"webmachine","1.9.1-rmq3.1.3-git52e62bc"},<br> {mochiweb,"MochiMedia Web Server","2.3.1-rmq3.1.3-gitd541e9a"},<br> {xmerl,"XML parser","1.2.10"},<br> {inets,"INETS CXC 138 49","5.7.1"},<br> {mnesia,"MNESIA CXC 138 12","4.5"},<br> {amqp_client,"RabbitMQ AMQP Client","3.1.3"},<br> {sasl,"SASL CXC 138 11","2.1.10"},<br> {stdlib,"ERTS CXC 138 10","1.17.5"},<br> {kernel,"ERTS CXC 138 10","2.14.5"}]},<br> {os,{unix,linux}},<br> {erlang_version,<br> "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},<br> {memory,<br> [{total,33671216},<br> {connection_procs,5408},<br> {queue_procs,5408},<br> {plugins,452592},<br> {other_proc,10742928},<br> {mnesia,61984},<br> {mgmt_db,10248},<br> {msg_index,34160},<br> {other_ets,1117456},<br> {binary,35464},<br> {code,17740702},<br> {atom,1602625},<br> {other_system,1862241}]},<br> {vm_memory_high_watermark,0.4},<br> {vm_memory_limit,153295257},<br> {disk_free_limit,1000000000},<br> {disk_free,77554319360},<br> {file_descriptors,<br> [{total_limit,924},{total_used,3},{sockets_limit,829},{sockets_used,1}]},<br> {processes,[{limit,1048576},{used,196}]},<br> {run_queue,0},<br> {uptime,335}]<br>Cluster status of node rabbit@qcluster0 ...<br>[{nodes,[{disc,[rabbit@qcluster0,rabbit@qcluster2]},{ram,[rabbit@qcluster1]}]},<br> {running_nodes,[rabbit@qcluster1,rabbit@qcluster2,rabbit@qcluster0]},<br> {partitions,[]}]<br>Application environment of node rabbit@qcluster0 ...<br>[{auth_backends,[rabbit_auth_backend_internal]},<br> {auth_mechanisms,['PLAIN','AMQPLAIN']},<br> {backing_queue_module,rabbit_variable_queue},<br> {cluster_nodes,{[rabbit@qcluster0,rabbit@qcluster1],disc}},<br> {cluster_partition_handling,pause_minority},<br> {collect_statistics,fine},<br> {collect_statistics_interval,5000},<br> {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},<br> {default_user,<<"guest">>},<br> {default_user_tags,[administrator]},<br> {default_vhost,<<"/">>},<br> {delegate_count,16},<br> {disk_free_limit,1000000000},<br> {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},<br> {error_logger,{file,"/var/log/rabbitmq/rabbit@qcluster0.log"}},<br> {frame_max,131072},<br> {heartbeat,600},<br> {hipe_compile,false},<br> {included_applications,[]},<br> {log_levels,[{connection,info}]},<br> {msg_store_file_size_limit,16777216},<br> {msg_store_index_module,rabbit_msg_store_ets_index},<br> {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.3/sbin/../plugins"},<br> {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@qcluster0-plugins-expand"},<br> {queue_index_max_journal_entries,65536},<br> {reverse_dns_lookups,false},<br> {sasl_error_logger,{file,"/var/log/rabbitmq/rabbit@qcluster0-sasl.log"}},<br> {server_properties,[]},<br> {ssl_cert_login_from,distinguished_name},<br> {ssl_listeners,[]},<br> {ssl_options,[]},<br> {tcp_listen_options,[binary,<br> {packet,raw},<br> {reuseaddr,true},<br> {backlog,128},<br> {nodelay,true},<br> {linger,{true,0}},<br> {exit_on_close,false}]},<br> {tcp_listeners,[{"auto",5672}]},<br> {trace_vhosts,[]},<br> {vm_memory_high_watermark,0.4}]<br>Connections:<br>Channels:<br>Queues on /:<br>Exchanges on /:<br>name<span class="Apple-tab-span" style="white-space:pre"> </span>type<span class="Apple-tab-span" style="white-space:pre"> </span>durable<span class="Apple-tab-span" style="white-space:pre"> </span>auto_delete<span class="Apple-tab-span" style="white-space:pre"> </span>internal<span class="Apple-tab-span" style="white-space:pre"> </span>arguments<span class="Apple-tab-span" style="white-space:pre"> </span>policy<br><span class="Apple-tab-span" style="white-space:pre"> </span>direct<span class="Apple-tab-span" style="white-space:pre"> </span>true<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>[]<br>amq.direct<span class="Apple-tab-span" style="white-space:pre"> </span>direct<span class="Apple-tab-span" style="white-space:pre"> </span>true<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>[]<br>amq.fanout<span class="Apple-tab-span" style="white-space:pre"> </span>fanout<span class="Apple-tab-span" style="white-space:pre"> </span>true<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>[]<br>amq.headers<span class="Apple-tab-span" style="white-space:pre"> </span>headers<span class="Apple-tab-span" style="white-space:pre"> </span>true<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>[]<br>amq.match<span class="Apple-tab-span" style="white-space:pre"> </span>headers<span class="Apple-tab-span" style="white-space:pre"> </span>true<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>[]<br>amq.rabbitmq.log<span class="Apple-tab-span" style="white-space:pre"> </span>topic<span class="Apple-tab-span" style="white-space:pre"> </span>true<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>[]<br>amq.rabbitmq.trace<span class="Apple-tab-span" style="white-space:pre"> </span>topic<span class="Apple-tab-span" style="white-space:pre"> </span>true<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>[]<br>amq.topic<span class="Apple-tab-span" style="white-space:pre"> </span>topic<span class="Apple-tab-span" style="white-space:pre"> </span>true<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>false<span class="Apple-tab-span" style="white-space:pre"> </span>[]<br>Bindings on /:<br>Consumers on /:<br>Permissions on /:<br>user<span class="Apple-tab-span" style="white-space:pre"> </span>configure<span class="Apple-tab-span" style="white-space:pre"> </span>write<span class="Apple-tab-span" style="white-space:pre"> </span>read<br>guest<span class="Apple-tab-span" style="white-space:pre"> </span>.*<span class="Apple-tab-span" style="white-space:pre"> </span>.*<span class="Apple-tab-span" style="white-space:pre"> </span>.*<br>Policies on /:<br>Parameters on /:<br>...done. </blockquote><div>Any help would be appreciated. </div><div>Thanks </div>
</div>