I ran this with 4 nodes in the cluster and after just 2 nodes down, the whole system became unresponsive.<div>I have put the log in here: https://gist.github.com/elbuo8/e5171ec85608b7bd7842</div><div><br></div><div>Also, I noticed that when I bring the nodes back up, they become Disc nodes even when they have been configured specifically as RAM nodes.</div><div><br><br>On Tuesday, August 6, 2013 11:47:31 AM UTC-4, Yamil Einar Asusta Santos wrote:<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;">I have been testing my cluster and I have come across an unexpected behavior. As explained in the subject, the cluster runs well on 3 nodes. If I bring 1 node down, it still runs smoothly. But if I bring a second node down, the third one becomes unresponsive.<div>This is what the third node provides after running "rabbitmqctl report":</div><div><br></div><div>
<blockquote>Reporting server status on {{2013,8,6},{15,19,8}}<br> ...<br>Error: {aborted,{no_exists,rabbit_<wbr>vhost}}</blockquote><div><br></div><div>Then if I bring 1 or 2 of my nodes back up, the third one becomes responsive and everything is back to normal. </div><div><br></div><div>Here is the report after bringing the nodes up again:</div><div><br></div><blockquote>Reporting server status on {{2013,8,6},{15,26,31}}<br> ...<br>Status of node rabbit@qcluster1 ...<br>[{pid,901},<br> {running_applications,<br> [{rabbitmq_management,"<wbr>RabbitMQ Management Console","3.1.3"},<br> {rabbitmq_management_agent,"<wbr>RabbitMQ Management Agent","3.1.3"},<br> {rabbit,"RabbitMQ","3.1.3"},<br> {os_mon,"CPO CXC 138 46","2.2.7"},<br> {rabbitmq_web_dispatch,"<wbr>RabbitMQ Web Dispatcher","3.1.3"},<br> {webmachine,"webmachine","1.9.<wbr>1-rmq3.1.3-git52e62bc"},<br> {mochiweb,"MochiMedia Web Server","2.3.1-rmq3.1.3-<wbr>gitd541e9a"},<br> {xmerl,"XML parser","1.2.10"},<br> {inets,"INETS CXC 138 49","5.7.1"},<br> {mnesia,"MNESIA CXC 138 12","4.5"},<br> {amqp_client,"RabbitMQ AMQP Client","3.1.3"},<br> {sasl,"SASL CXC 138 11","2.1.10"},<br> {stdlib,"ERTS CXC 138 10","1.17.5"},<br> {kernel,"ERTS CXC 138 10","2.14.5"}]},<br> {os,{unix,linux}},<br> {erlang_version,<br> "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},<br> {memory,<br> [{total,31689208},<br> {connection_procs,5408},<br> {queue_procs,5408},<br> {plugins,340592},<br> {other_proc,9166952},<br> {mnesia,61984},<br> {mgmt_db,10248},<br> {msg_index,34160},<br> {other_ets,1080168},<br> {binary,6480},<br> {code,17557932},<br> {atom,1565809},<br> {other_system,1854067}]},<br> {vm_memory_high_watermark,0.<wbr>4},<br> {vm_memory_limit,153295257},<br> {disk_free_limit,1000000000},<br> {disk_free,77554466816},<br> {file_descriptors,<br> [{total_limit,924},{total_<wbr>used,3},{sockets_limit,829},{<wbr>sockets_used,1}]},<br> {processes,[{limit,1048576},{<wbr>used,187}]},<br> {run_queue,0},<br> {uptime,210}]<br>Cluster status of node rabbit@qcluster1 ...<br>[{nodes,[{disc,[rabbit@<wbr>qcluster2,rabbit@qcluster0]},{<wbr>ram,[rabbit@qcluster1]}]},<br> {running_nodes,[rabbit@<wbr>qcluster0,rabbit@qcluster2,<wbr>rabbit@qcluster1]},<br> {partitions,[]}]<br>Application environment of node rabbit@qcluster1 ...<br>[{auth_backends,[rabbit_auth_<wbr>backend_internal]},<br> {auth_mechanisms,['PLAIN','<wbr>AMQPLAIN']},<br> {backing_queue_module,rabbit_<wbr>variable_queue},<br> {cluster_nodes,{[rabbit@<wbr>qcluster0,rabbit@qcluster1],<wbr>ram}},<br> {cluster_partition_handling,<wbr>pause_minority},<br> {collect_statistics,fine},<br> {collect_statistics_interval,<wbr>5000},<br> {default_permissions,[<<".*"><wbr>>,<<".*">>,<<".*">>]},<br> {default_user,<<"guest">>},<br> {default_user_tags,[<wbr>administrator]},<br> {default_vhost,<<"/">>},<br> {delegate_count,16},<br> {disk_free_limit,1000000000},<br> {enabled_plugins_file,"/etc/<wbr>rabbitmq/enabled_plugins"},<br> {error_logger,{file,"/var/<wbr>log/rabbitmq/rabbit@qcluster1.<wbr>log"}},<br> {frame_max,131072},<br> {heartbeat,600},<br> {hipe_compile,false},<br> {included_applications,[]},<br> {log_levels,[{connection,<wbr>info}]},<br> {msg_store_file_size_limit,<wbr>16777216},<br> {msg_store_index_module,<wbr>rabbit_msg_store_ets_index},<br> {plugins_dir,"/usr/lib/<wbr>rabbitmq/lib/rabbitmq_server-<wbr>3.1.3/sbin/../plugins"},<br> {plugins_expand_dir,"/var/<wbr>lib/rabbitmq/mnesia/rabbit@<wbr>qcluster1-plugins-expand"},<br> {queue_index_max_journal_<wbr>entries,65536},<br> {reverse_dns_lookups,false},<br> {sasl_error_logger,{file,"/<wbr>var/log/rabbitmq/rabbit@<wbr>qcluster1-sasl.log"}},<br> {server_properties,[]},<br> {ssl_cert_login_from,<wbr>distinguished_name},<br> {ssl_listeners,[]},<br> {ssl_options,[]},<br> {tcp_listen_options,[binary,<br> {packet,raw},<br> {reuseaddr,true},<br> {backlog,128},<br> {nodelay,true},<br> {linger,{true,0}},<br> {exit_on_close,false}]},<br> {tcp_listeners,[{"auto",5672}<wbr>]},<br> {trace_vhosts,[]},<br> {vm_memory_high_watermark,0.<wbr>4}]<br>Status of node rabbit@qcluster2 ...<br>[{pid,1940},<br> {running_applications,<br> [{rabbitmq_management,"<wbr>RabbitMQ Management Console","3.1.3"},<br> {rabbitmq_management_agent,"<wbr>RabbitMQ Management Agent","3.1.3"},<br> {rabbit,"RabbitMQ","3.1.3"},<br> {os_mon,"CPO CXC 138 46","2.2.7"},<br> {rabbitmq_web_dispatch,"<wbr>RabbitMQ Web Dispatcher","3.1.3"},<br> {webmachine,"webmachine","1.9.<wbr>1-rmq3.1.3-git52e62bc"},<br> {mochiweb,"MochiMedia Web Server","2.3.1-rmq3.1.3-<wbr>gitd541e9a"},<br> {mnesia,"MNESIA CXC 138 12","4.5"},<br> {amqp_client,"RabbitMQ AMQP Client","3.1.3"},<br> {xmerl,"XML parser","1.2.10"},<br> {inets,"INETS CXC 138 49","5.7.1"},<br> {sasl,"SASL CXC 138 11","2.1.10"},<br> {stdlib,"ERTS CXC 138 10","1.17.5"},<br> {kernel,"ERTS CXC 138 10","2.14.5"}]},<br> {os,{unix,linux}},<br> {erlang_version,<br> "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},<br> {memory,<br> [{total,32032800},<br> {connection_procs,5408},<br> {queue_procs,5408},<br> {plugins,315040},<br> {other_proc,9172832},<br> {mnesia,61952},<br> {mgmt_db,89704},<br> {msg_index,34160},<br> {other_ets,1111736},<br> {binary,33384},<br> {code,17743902},<br> {atom,1603433},<br> {other_system,1855841}]},<br> {vm_memory_high_watermark,0.<wbr>4},<br> {vm_memory_limit,153295257},<br> {disk_free_limit,1000000000},<br> {disk_free,77553033216},<br> {file_descriptors,<br> [{total_limit,924},{total_<wbr>used,3},{sockets_limit,829},{<wbr>sockets_used,1}]},<br> {processes,[{limit,1048576},{<wbr>used,191}]},<br> {run_queue,0},<br> {uptime,617}]<br>Cluster status of node rabbit@qcluster2 ...<br>[{nodes,[{disc,[rabbit@<wbr>qcluster0,rabbit@qcluster2]},{<wbr>ram,[rabbit@qcluster1]}]},<br> {running_nodes,[rabbit@<wbr>qcluster1,rabbit@qcluster0,<wbr>rabbit@qcluster2]},<br> {partitions,[]}]<br>Application environment of node rabbit@qcluster2 ...<br>[{auth_backends,[rabbit_auth_<wbr>backend_internal]},<br> {auth_mechanisms,['PLAIN','<wbr>AMQPLAIN']},<br> {backing_queue_module,rabbit_<wbr>variable_queue},<br> {cluster_nodes,{[rabbit@<wbr>qcluster0,rabbit@qcluster1,<wbr>rabbit@qcluster2],ram}},<br> {cluster_partition_handling,<wbr>pause_minority},<br> {collect_statistics,fine},<br> {collect_statistics_interval,<wbr>5000},<br> {default_permissions,[<<".*"><wbr>>,<<".*">>,<<".*">>]},<br> {default_user,<<"guest">>},<br> {default_user_tags,[<wbr>administrator]},<br> {default_vhost,<<"/">>},<br> {delegate_count,16},<br> {disk_free_limit,1000000000},<br> {enabled_plugins_file,"/etc/<wbr>rabbitmq/enabled_plugins"},<br> {error_logger,{file,"/var/<wbr>log/rabbitmq/rabbit@qcluster2.<wbr>log"}},<br> {frame_max,131072},<br> {heartbeat,600},<br> {hipe_compile,false},<br> {included_applications,[]},<br> {log_levels,[{connection,<wbr>info}]},<br> {msg_store_file_size_limit,<wbr>16777216},<br> {msg_store_index_module,<wbr>rabbit_msg_store_ets_index},<br> {plugins_dir,"/usr/lib/<wbr>rabbitmq/lib/rabbitmq_server-<wbr>3.1.3/sbin/../plugins"},<br> {plugins_expand_dir,"/var/<wbr>lib/rabbitmq/mnesia/rabbit@<wbr>qcluster2-plugins-expand"},<br> {queue_index_max_journal_<wbr>entries,65536},<br> {reverse_dns_lookups,false},<br> {sasl_error_logger,{file,"/<wbr>var/log/rabbitmq/rabbit@<wbr>qcluster2-sasl.log"}},<br> {server_properties,[]},<br> {ssl_cert_login_from,<wbr>distinguished_name},<br> {ssl_listeners,[]},<br> {ssl_options,[]},<br> {tcp_listen_options,[binary,<br> {packet,raw},<br> {reuseaddr,true},<br> {backlog,128},<br> {nodelay,true},<br> {linger,{true,0}},<br> {exit_on_close,false}]},<br> {tcp_listeners,[{"auto",5672}<wbr>]},<br> {trace_vhosts,[]},<br> {vm_memory_high_watermark,0.<wbr>4}]<br>Status of node rabbit@qcluster0 ...<br>[{pid,873},<br> {running_applications,<br> [{rabbitmq_management,"<wbr>RabbitMQ Management Console","3.1.3"},<br> {rabbitmq_management_agent,"<wbr>RabbitMQ Management Agent","3.1.3"},<br> {rabbit,"RabbitMQ","3.1.3"},<br> {os_mon,"CPO CXC 138 46","2.2.7"},<br> {rabbitmq_web_dispatch,"<wbr>RabbitMQ Web Dispatcher","3.1.3"},<br> {webmachine,"webmachine","1.9.<wbr>1-rmq3.1.3-git52e62bc"},<br> {mochiweb,"MochiMedia Web Server","2.3.1-rmq3.1.3-<wbr>gitd541e9a"},<br> {xmerl,"XML parser","1.2.10"},<br> {inets,"INETS CXC 138 49","5.7.1"},<br> {mnesia,"MNESIA CXC 138 12","4.5"},<br> {amqp_client,"RabbitMQ AMQP Client","3.1.3"},<br> {sasl,"SASL CXC 138 11","2.1.10"},<br> {stdlib,"ERTS CXC 138 10","1.17.5"},<br> {kernel,"ERTS CXC 138 10","2.14.5"}]},<br> {os,{unix,linux}},<br> {erlang_version,<br> "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},<br> {memory,<br> [{total,33671216},<br> {connection_procs,5408},<br> {queue_procs,5408},<br> {plugins,452592},<br> {other_proc,10742928},<br> {mnesia,61984},<br> {mgmt_db,10248},<br> {msg_index,34160},<br> {other_ets,1117456},<br> {binary,35464},<br> {code,17740702},<br> {atom,1602625},<br> {other_system,1862241}]},<br> {vm_memory_high_watermark,0.<wbr>4},<br> {vm_memory_limit,153295257},<br> {disk_free_limit,1000000000},<br> {disk_free,77554319360},<br> {file_descriptors,<br> [{total_limit,924},{total_<wbr>used,3},{sockets_limit,829},{<wbr>sockets_used,1}]},<br> {processes,[{limit,1048576},{<wbr>used,196}]},<br> {run_queue,0},<br> {uptime,335}]<br>Cluster status of node rabbit@qcluster0 ...<br>[{nodes,[{disc,[rabbit@<wbr>qcluster0,rabbit@qcluster2]},{<wbr>ram,[rabbit@qcluster1]}]},<br> {running_nodes,[rabbit@<wbr>qcluster1,rabbit@qcluster2,<wbr>rabbit@qcluster0]},<br> {partitions,[]}]<br>Application environment of node rabbit@qcluster0 ...<br>[{auth_backends,[rabbit_auth_<wbr>backend_internal]},<br> {auth_mechanisms,['PLAIN','<wbr>AMQPLAIN']},<br> {backing_queue_module,rabbit_<wbr>variable_queue},<br> {cluster_nodes,{[rabbit@<wbr>qcluster0,rabbit@qcluster1],<wbr>disc}},<br> {cluster_partition_handling,<wbr>pause_minority},<br> {collect_statistics,fine},<br> {collect_statistics_interval,<wbr>5000},<br> {default_permissions,[<<".*"><wbr>>,<<".*">>,<<".*">>]},<br> {default_user,<<"guest">>},<br> {default_user_tags,[<wbr>administrator]},<br> {default_vhost,<<"/">>},<br> {delegate_count,16},<br> {disk_free_limit,1000000000},<br> {enabled_plugins_file,"/etc/<wbr>rabbitmq/enabled_plugins"},<br> {error_logger,{file,"/var/<wbr>log/rabbitmq/rabbit@qcluster0.<wbr>log"}},<br> {frame_max,131072},<br> {heartbeat,600},<br> {hipe_compile,false},<br> {included_applications,[]},<br> {log_levels,[{connection,<wbr>info}]},<br> {msg_store_file_size_limit,<wbr>16777216},<br> {msg_store_index_module,<wbr>rabbit_msg_store_ets_index},<br> {plugins_dir,"/usr/lib/<wbr>rabbitmq/lib/rabbitmq_server-<wbr>3.1.3/sbin/../plugins"},<br> {plugins_expand_dir,"/var/<wbr>lib/rabbitmq/mnesia/rabbit@<wbr>qcluster0-plugins-expand"},<br> {queue_index_max_journal_<wbr>entries,65536},<br> {reverse_dns_lookups,false},<br> {sasl_error_logger,{file,"/<wbr>var/log/rabbitmq/rabbit@<wbr>qcluster0-sasl.log"}},<br> {server_properties,[]},<br> {ssl_cert_login_from,<wbr>distinguished_name},<br> {ssl_listeners,[]},<br> {ssl_options,[]},<br> {tcp_listen_options,[binary,<br> {packet,raw},<br> {reuseaddr,true},<br> {backlog,128},<br> {nodelay,true},<br> {linger,{true,0}},<br> {exit_on_close,false}]},<br> {tcp_listeners,[{"auto",5672}<wbr>]},<br> {trace_vhosts,[]},<br> {vm_memory_high_watermark,0.<wbr>4}]<br>Connections:<br>Channels:<br>Queues on /:<br>Exchanges on /:<br>name<span style="white-space:pre"> </span>type<span style="white-space:pre"> </span>durable<span style="white-space:pre"> </span>auto_delete<span style="white-space:pre"> </span>internal<span style="white-space:pre"> </span>arguments<span style="white-space:pre"> </span>policy<br><span style="white-space:pre"> </span>direct<span style="white-space:pre"> </span>true<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>[]<br>amq.direct<span style="white-space:pre"> </span>direct<span style="white-space:pre"> </span>true<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>[]<br>amq.fanout<span style="white-space:pre"> </span>fanout<span style="white-space:pre"> </span>true<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>[]<br>amq.headers<span style="white-space:pre"> </span>headers<span style="white-space:pre"> </span>true<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>[]<br>amq.match<span style="white-space:pre"> </span>headers<span style="white-space:pre"> </span>true<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>[]<br>amq.rabbitmq.log<span style="white-space:pre"> </span>topic<span style="white-space:pre"> </span>true<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>[]<br>amq.rabbitmq.trace<span style="white-space:pre"> </span>topic<span style="white-space:pre"> </span>true<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>[]<br>amq.topic<span style="white-space:pre"> </span>topic<span style="white-space:pre"> </span>true<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>false<span style="white-space:pre"> </span>[]<br>Bindings on /:<br>Consumers on /:<br>Permissions on /:<br>user<span style="white-space:pre"> </span>configure<span style="white-space:pre"> </span>write<span style="white-space:pre"> </span>read<br>guest<span style="white-space:pre"> </span>.*<span style="white-space:pre"> </span>.*<span style="white-space:pre"> </span>.*<br>Policies on /:<br>Parameters on /:<br>...done. </blockquote><div>Any help would be appreciated. </div><div>Thanks </div>
</div></blockquote></div>