Hi Matthew,<div><br></div><div>Thanks a lot for the reply.</div><div><br></div><div>I am not sure why this error still appears after I attempt &#39;rabbitmqctl start_app&#39;. I tried disk clustering as well, with the same result. The error logs look like this:</div>


<div><br></div><div>rabbit@node117.log:  <a href="http://pastebin.com/embhVaea" target="_blank">http://pastebin.com/embhVaea</a></div><div>rabbit@node117.log.1: <a href="http://pastebin.com/Xa5ZnWrC" target="_blank">http://pastebin.com/Xa5ZnWrC</a></div>


<div>rabbit@node117-sasl.log: (empty)</div><div>rabbit@node117-sasl.log.1: <a href="http://pastebin.com/Au745uNU" target="_blank">http://pastebin.com/Au745uNU</a></div>
<div><br></div><div>node117 is the proxy node, with which I cluster all other 99 nodes (as either disc or ram nodes). This seems to be the error (from the logs): </div><div><br>exception exit: {{badmatch,<br>                         {error,<br>


                             {file_error,<br>                                 &quot;/var/lib/rabbitmq/mnesia/rabbit@node117/msg_store_transient/file_summary.ets&quot;,<br>                                 enoent}}},</div>

<div>
<br></div><div>Regards,</div><div><br><div class="gmail_quote">2011/3/28 Matthew Sackman <span dir="ltr">&lt;<a href="mailto:matthew@rabbitmq.com" target="_blank">matthew@rabbitmq.com</a>&gt;</span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Advait,<br>
<div><br>
On Sun, Mar 27, 2011 at 11:01:53AM +0530, Advait Alai wrote:<br>
&gt; I&#39;m trying to add 150 nodes to a RabbitMQ cluster. After around 50 nodes,<br>
&gt; the stop-reset-cluster-start iteration starts giving the error:<br>
&gt;<br>
&gt; Stopping node rabbit@node130 ...<br>
&gt; ...done.<br>
&gt; Resetting node rabbit@node130 ...<br>
&gt; ...done.<br>
&gt; Clustering node rabbit@node130 with [rabbit@node117] ...<br>
&gt; ...done.<br>
&gt; Starting node rabbit@node130 ...<br>
&gt; Error: {cannot_start_application,rabbit,<br>
&gt;            {bad_return,<br>
&gt;                {{rabbit,start,[normal,[]]},<br>
&gt;                 {&#39;EXIT&#39;,{rabbit,failure_during_boot}}}}}<br>
<br>
</div>Hmm, curious. I don&#39;t know why that happens. I&#39;ve just successfully<br>
built a cluster of 100 nodes. I&#39;m building from source, so using the<br>
Makefile. I did:<br>
<br>
1. Edit the Makefile. Around line 153, remove the<br>
                RABBITMQ_ALLOW_INPUT=true \<br>
  line completely.<br>
<br>
2. rabbitmq-server$ for t in $(seq 1 100); do make RABBITMQ_NODENAME=rabbit_$t RABBITMQ_NODE_PORT=$(echo &quot;5672+$t&quot;|bc) TMPDIR=/home/matthew/ssd/ cleandb run &amp; done<br>
<br>
3. rabbitmq-server/scripts$ for t in $(seq 2 100); do ./rabbitmqctl -n rabbit_$t@hazel stop_app ; ./rabbitmqctl -n rabbit_$t@hazel reset &amp;&amp; ./rabbitmqctl -n rabbit_$t@hazel cluster rabbit_1@hazel rabbit_$t@hazel &amp;&amp; ./rabbitmqctl -n rabbit_$t@hazel start_app; done<br>



<br>
4. Go make a cuppa tea.<br>
<br>
It takes a while, and slows down on a quadratic (which isn&#39;t surprising<br>
because we&#39;re forming an all-to-all mesh), and it eats up a lot of RAM.<br>
But it did get there for me:<br>
<br>
rabbitmq-server/scripts$ ./rabbitmqctl -n &#39;rabbit_1@hazel&#39; status<br>
Status of node rabbit_1@hazel ...<br>
[{pid,7933},<br>
 {running_applications,[{rabbit,&quot;RabbitMQ&quot;,&quot;%%VSN%%&quot;},<br>
                        {mnesia,&quot;MNESIA  CXC 138 12&quot;,&quot;4.4.17&quot;},<br>
                        {os_mon,&quot;CPO  CXC 138 46&quot;,&quot;2.2.5&quot;},<br>
                        {sasl,&quot;SASL  CXC 138 11&quot;,&quot;2.1.9.3&quot;},<br>
                        {stdlib,&quot;ERTS  CXC 138 10&quot;,&quot;1.17.3&quot;},<br>
                        {kernel,&quot;ERTS  CXC 138 10&quot;,&quot;2.14.3&quot;}]},<br>
 {nodes,[{disc,[rabbit_100@hazel,rabbit_99@hazel,rabbit_98@hazel,<br>
                rabbit_97@hazel,rabbit_96@hazel,rabbit_95@hazel,<br>
                rabbit_94@hazel,rabbit_93@hazel,rabbit_92@hazel,<br>
                rabbit_91@hazel,rabbit_90@hazel,rabbit_89@hazel,<br>
                rabbit_88@hazel,rabbit_87@hazel,rabbit_86@hazel,<br>
                rabbit_85@hazel,rabbit_84@hazel,rabbit_83@hazel,<br>
                rabbit_82@hazel,rabbit_81@hazel,rabbit_80@hazel,<br>
                rabbit_79@hazel,rabbit_78@hazel,rabbit_77@hazel,<br>
                rabbit_76@hazel,rabbit_75@hazel,rabbit_74@hazel,<br>
                rabbit_73@hazel,rabbit_72@hazel,rabbit_71@hazel,<br>
                rabbit_70@hazel,rabbit_69@hazel,rabbit_68@hazel,<br>
                rabbit_67@hazel,rabbit_66@hazel,rabbit_65@hazel,<br>
                rabbit_64@hazel,rabbit_63@hazel,rabbit_62@hazel,<br>
                rabbit_61@hazel,rabbit_60@hazel,rabbit_59@hazel,<br>
                rabbit_58@hazel,rabbit_57@hazel,rabbit_56@hazel,<br>
                rabbit_55@hazel,rabbit_54@hazel,rabbit_53@hazel,<br>
                rabbit_52@hazel,rabbit_51@hazel,rabbit_50@hazel,<br>
                rabbit_49@hazel,rabbit_48@hazel,rabbit_47@hazel,<br>
                rabbit_46@hazel,rabbit_45@hazel,rabbit_44@hazel,<br>
                rabbit_43@hazel,rabbit_42@hazel,rabbit_41@hazel,<br>
                rabbit_40@hazel,rabbit_39@hazel,rabbit_38@hazel,<br>
                rabbit_37@hazel,rabbit_36@hazel,rabbit_35@hazel,<br>
                rabbit_34@hazel,rabbit_33@hazel,rabbit_32@hazel,<br>
                rabbit_31@hazel,rabbit_30@hazel,rabbit_29@hazel,<br>
                rabbit_28@hazel,rabbit_27@hazel,rabbit_26@hazel,<br>
                rabbit_25@hazel,rabbit_24@hazel,rabbit_23@hazel,<br>
                rabbit_22@hazel,rabbit_21@hazel,rabbit_20@hazel,<br>
                rabbit_19@hazel,rabbit_18@hazel,rabbit_17@hazel,<br>
                rabbit_16@hazel,rabbit_15@hazel,rabbit_14@hazel,<br>
                rabbit_13@hazel,rabbit_12@hazel,rabbit_11@hazel,<br>
                rabbit_10@hazel,rabbit_9@hazel,rabbit_8@hazel,rabbit_7@hazel,<br>
                rabbit_6@hazel,rabbit_5@hazel,rabbit_4@hazel,rabbit_3@hazel,<br>
                rabbit_2@hazel,rabbit_1@hazel]}]},<br>
 {running_nodes,[rabbit_100@hazel,rabbit_99@hazel,rabbit_98@hazel,<br>
                 rabbit_97@hazel,rabbit_96@hazel,rabbit_95@hazel,<br>
                 rabbit_94@hazel,rabbit_93@hazel,rabbit_92@hazel,<br>
                 rabbit_91@hazel,rabbit_90@hazel,rabbit_89@hazel,<br>
                 rabbit_88@hazel,rabbit_87@hazel,rabbit_86@hazel,<br>
                 rabbit_85@hazel,rabbit_84@hazel,rabbit_83@hazel,<br>
                 rabbit_82@hazel,rabbit_81@hazel,rabbit_80@hazel,<br>
                 rabbit_79@hazel,rabbit_78@hazel,rabbit_77@hazel,<br>
                 rabbit_76@hazel,rabbit_75@hazel,rabbit_74@hazel,<br>
                 rabbit_73@hazel,rabbit_72@hazel,rabbit_71@hazel,<br>
                 rabbit_70@hazel,rabbit_69@hazel,rabbit_68@hazel,<br>
                 rabbit_67@hazel,rabbit_66@hazel,rabbit_65@hazel,<br>
                 rabbit_64@hazel,rabbit_63@hazel,rabbit_62@hazel,<br>
                 rabbit_61@hazel,rabbit_60@hazel,rabbit_59@hazel,<br>
                 rabbit_58@hazel,rabbit_57@hazel,rabbit_56@hazel,<br>
                 rabbit_55@hazel,rabbit_54@hazel,rabbit_53@hazel,<br>
                 rabbit_52@hazel,rabbit_51@hazel,rabbit_50@hazel,<br>
                 rabbit_49@hazel,rabbit_48@hazel,rabbit_47@hazel,<br>
                 rabbit_46@hazel,rabbit_45@hazel,rabbit_44@hazel,<br>
                 rabbit_43@hazel,rabbit_42@hazel,rabbit_41@hazel,<br>
                 rabbit_40@hazel,rabbit_39@hazel,rabbit_38@hazel,<br>
                 rabbit_37@hazel,rabbit_36@hazel,rabbit_35@hazel,<br>
                 rabbit_34@hazel,rabbit_33@hazel,rabbit_32@hazel,<br>
                 rabbit_31@hazel,rabbit_30@hazel,rabbit_29@hazel,<br>
                 rabbit_28@hazel,rabbit_27@hazel,rabbit_26@hazel,<br>
                 rabbit_25@hazel,rabbit_24@hazel,rabbit_23@hazel,<br>
                 rabbit_22@hazel,rabbit_21@hazel,rabbit_20@hazel,<br>
                 rabbit_19@hazel,rabbit_18@hazel,rabbit_17@hazel,<br>
                 rabbit_16@hazel,rabbit_15@hazel,rabbit_14@hazel,<br>
                 rabbit_13@hazel,rabbit_12@hazel,rabbit_11@hazel,<br>
                 rabbit_10@hazel,rabbit_9@hazel,rabbit_8@hazel,rabbit_7@hazel,<br>
                 rabbit_6@hazel,rabbit_5@hazel,rabbit_4@hazel,rabbit_3@hazel,<br>
                 rabbit_2@hazel,rabbit_1@hazel]}]<br>
...done.<br>
<font color="#888888"><br>
Matthew<br>
</font><div><div></div><div>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br></div>