Hi Matthew,<div><br></div><div>Thanks a lot for the reply.</div><div><br></div><div>I am not sure why this error still appears after I attempt 'rabbitmqctl start_app'. I tried disk clustering as well, with the same result. The error logs look like this:</div>
<div><br></div><div>rabbit@node117.log: <a href="http://pastebin.com/embhVaea" target="_blank">http://pastebin.com/embhVaea</a></div><div>rabbit@node117.log.1: <a href="http://pastebin.com/Xa5ZnWrC" target="_blank">http://pastebin.com/Xa5ZnWrC</a></div>
<div>rabbit@node117-sasl.log: (empty)</div><div>rabbit@node117-sasl.log.1: <a href="http://pastebin.com/Au745uNU" target="_blank">http://pastebin.com/Au745uNU</a></div>
<div><br></div><div>node117 is the proxy node, with which I cluster all other 99 nodes (as either disc or ram nodes). This seems to be the error (from the logs): </div><div><br>exception exit: {{badmatch,<br> {error,<br>
{file_error,<br> "/var/lib/rabbitmq/mnesia/rabbit@node117/msg_store_transient/file_summary.ets",<br> enoent}}},</div>
<div>
<br></div><div>Regards,</div><div><br><div class="gmail_quote">2011/3/28 Matthew Sackman <span dir="ltr"><<a href="mailto:matthew@rabbitmq.com" target="_blank">matthew@rabbitmq.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Advait,<br>
<div><br>
On Sun, Mar 27, 2011 at 11:01:53AM +0530, Advait Alai wrote:<br>
> I'm trying to add 150 nodes to a RabbitMQ cluster. After around 50 nodes,<br>
> the stop-reset-cluster-start iteration starts giving the error:<br>
><br>
> Stopping node rabbit@node130 ...<br>
> ...done.<br>
> Resetting node rabbit@node130 ...<br>
> ...done.<br>
> Clustering node rabbit@node130 with [rabbit@node117] ...<br>
> ...done.<br>
> Starting node rabbit@node130 ...<br>
> Error: {cannot_start_application,rabbit,<br>
> {bad_return,<br>
> {{rabbit,start,[normal,[]]},<br>
> {'EXIT',{rabbit,failure_during_boot}}}}}<br>
<br>
</div>Hmm, curious. I don't know why that happens. I've just successfully<br>
built a cluster of 100 nodes. I'm building from source, so using the<br>
Makefile. I did:<br>
<br>
1. Edit the Makefile. Around line 153, remove the<br>
RABBITMQ_ALLOW_INPUT=true \<br>
line completely.<br>
<br>
2. rabbitmq-server$ for t in $(seq 1 100); do make RABBITMQ_NODENAME=rabbit_$t RABBITMQ_NODE_PORT=$(echo "5672+$t"|bc) TMPDIR=/home/matthew/ssd/ cleandb run & done<br>
<br>
3. rabbitmq-server/scripts$ for t in $(seq 2 100); do ./rabbitmqctl -n rabbit_$t@hazel stop_app ; ./rabbitmqctl -n rabbit_$t@hazel reset && ./rabbitmqctl -n rabbit_$t@hazel cluster rabbit_1@hazel rabbit_$t@hazel && ./rabbitmqctl -n rabbit_$t@hazel start_app; done<br>
<br>
4. Go make a cuppa tea.<br>
<br>
It takes a while, and slows down on a quadratic (which isn't surprising<br>
because we're forming an all-to-all mesh), and it eats up a lot of RAM.<br>
But it did get there for me:<br>
<br>
rabbitmq-server/scripts$ ./rabbitmqctl -n 'rabbit_1@hazel' status<br>
Status of node rabbit_1@hazel ...<br>
[{pid,7933},<br>
{running_applications,[{rabbit,"RabbitMQ","%%VSN%%"},<br>
{mnesia,"MNESIA CXC 138 12","4.4.17"},<br>
{os_mon,"CPO CXC 138 46","2.2.5"},<br>
{sasl,"SASL CXC 138 11","2.1.9.3"},<br>
{stdlib,"ERTS CXC 138 10","1.17.3"},<br>
{kernel,"ERTS CXC 138 10","2.14.3"}]},<br>
{nodes,[{disc,[rabbit_100@hazel,rabbit_99@hazel,rabbit_98@hazel,<br>
rabbit_97@hazel,rabbit_96@hazel,rabbit_95@hazel,<br>
rabbit_94@hazel,rabbit_93@hazel,rabbit_92@hazel,<br>
rabbit_91@hazel,rabbit_90@hazel,rabbit_89@hazel,<br>
rabbit_88@hazel,rabbit_87@hazel,rabbit_86@hazel,<br>
rabbit_85@hazel,rabbit_84@hazel,rabbit_83@hazel,<br>
rabbit_82@hazel,rabbit_81@hazel,rabbit_80@hazel,<br>
rabbit_79@hazel,rabbit_78@hazel,rabbit_77@hazel,<br>
rabbit_76@hazel,rabbit_75@hazel,rabbit_74@hazel,<br>
rabbit_73@hazel,rabbit_72@hazel,rabbit_71@hazel,<br>
rabbit_70@hazel,rabbit_69@hazel,rabbit_68@hazel,<br>
rabbit_67@hazel,rabbit_66@hazel,rabbit_65@hazel,<br>
rabbit_64@hazel,rabbit_63@hazel,rabbit_62@hazel,<br>
rabbit_61@hazel,rabbit_60@hazel,rabbit_59@hazel,<br>
rabbit_58@hazel,rabbit_57@hazel,rabbit_56@hazel,<br>
rabbit_55@hazel,rabbit_54@hazel,rabbit_53@hazel,<br>
rabbit_52@hazel,rabbit_51@hazel,rabbit_50@hazel,<br>
rabbit_49@hazel,rabbit_48@hazel,rabbit_47@hazel,<br>
rabbit_46@hazel,rabbit_45@hazel,rabbit_44@hazel,<br>
rabbit_43@hazel,rabbit_42@hazel,rabbit_41@hazel,<br>
rabbit_40@hazel,rabbit_39@hazel,rabbit_38@hazel,<br>
rabbit_37@hazel,rabbit_36@hazel,rabbit_35@hazel,<br>
rabbit_34@hazel,rabbit_33@hazel,rabbit_32@hazel,<br>
rabbit_31@hazel,rabbit_30@hazel,rabbit_29@hazel,<br>
rabbit_28@hazel,rabbit_27@hazel,rabbit_26@hazel,<br>
rabbit_25@hazel,rabbit_24@hazel,rabbit_23@hazel,<br>
rabbit_22@hazel,rabbit_21@hazel,rabbit_20@hazel,<br>
rabbit_19@hazel,rabbit_18@hazel,rabbit_17@hazel,<br>
rabbit_16@hazel,rabbit_15@hazel,rabbit_14@hazel,<br>
rabbit_13@hazel,rabbit_12@hazel,rabbit_11@hazel,<br>
rabbit_10@hazel,rabbit_9@hazel,rabbit_8@hazel,rabbit_7@hazel,<br>
rabbit_6@hazel,rabbit_5@hazel,rabbit_4@hazel,rabbit_3@hazel,<br>
rabbit_2@hazel,rabbit_1@hazel]}]},<br>
{running_nodes,[rabbit_100@hazel,rabbit_99@hazel,rabbit_98@hazel,<br>
rabbit_97@hazel,rabbit_96@hazel,rabbit_95@hazel,<br>
rabbit_94@hazel,rabbit_93@hazel,rabbit_92@hazel,<br>
rabbit_91@hazel,rabbit_90@hazel,rabbit_89@hazel,<br>
rabbit_88@hazel,rabbit_87@hazel,rabbit_86@hazel,<br>
rabbit_85@hazel,rabbit_84@hazel,rabbit_83@hazel,<br>
rabbit_82@hazel,rabbit_81@hazel,rabbit_80@hazel,<br>
rabbit_79@hazel,rabbit_78@hazel,rabbit_77@hazel,<br>
rabbit_76@hazel,rabbit_75@hazel,rabbit_74@hazel,<br>
rabbit_73@hazel,rabbit_72@hazel,rabbit_71@hazel,<br>
rabbit_70@hazel,rabbit_69@hazel,rabbit_68@hazel,<br>
rabbit_67@hazel,rabbit_66@hazel,rabbit_65@hazel,<br>
rabbit_64@hazel,rabbit_63@hazel,rabbit_62@hazel,<br>
rabbit_61@hazel,rabbit_60@hazel,rabbit_59@hazel,<br>
rabbit_58@hazel,rabbit_57@hazel,rabbit_56@hazel,<br>
rabbit_55@hazel,rabbit_54@hazel,rabbit_53@hazel,<br>
rabbit_52@hazel,rabbit_51@hazel,rabbit_50@hazel,<br>
rabbit_49@hazel,rabbit_48@hazel,rabbit_47@hazel,<br>
rabbit_46@hazel,rabbit_45@hazel,rabbit_44@hazel,<br>
rabbit_43@hazel,rabbit_42@hazel,rabbit_41@hazel,<br>
rabbit_40@hazel,rabbit_39@hazel,rabbit_38@hazel,<br>
rabbit_37@hazel,rabbit_36@hazel,rabbit_35@hazel,<br>
rabbit_34@hazel,rabbit_33@hazel,rabbit_32@hazel,<br>
rabbit_31@hazel,rabbit_30@hazel,rabbit_29@hazel,<br>
rabbit_28@hazel,rabbit_27@hazel,rabbit_26@hazel,<br>
rabbit_25@hazel,rabbit_24@hazel,rabbit_23@hazel,<br>
rabbit_22@hazel,rabbit_21@hazel,rabbit_20@hazel,<br>
rabbit_19@hazel,rabbit_18@hazel,rabbit_17@hazel,<br>
rabbit_16@hazel,rabbit_15@hazel,rabbit_14@hazel,<br>
rabbit_13@hazel,rabbit_12@hazel,rabbit_11@hazel,<br>
rabbit_10@hazel,rabbit_9@hazel,rabbit_8@hazel,rabbit_7@hazel,<br>
rabbit_6@hazel,rabbit_5@hazel,rabbit_4@hazel,rabbit_3@hazel,<br>
rabbit_2@hazel,rabbit_1@hazel]}]<br>
...done.<br>
<font color="#888888"><br>
Matthew<br>
</font><div><div></div><div>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br></div>