[rabbitmq-discuss] Cluster size limit
Advait Alai
advaitalai at gmail.com
Sun Apr 3 12:58:44 BST 2011
Hi Matthew,
Thanks a lot for the reply.
I am not sure why this error still appears after I attempt 'rabbitmqctl
start_app'. I tried disk clustering as well, with the same result. The error
logs look like this:
rabbit at node117.log: http://pastebin.com/embhVaea
rabbit at node117.log.1: http://pastebin.com/Xa5ZnWrC
rabbit at node117-sasl.log: (empty)
rabbit at node117-sasl.log.1: http://pastebin.com/Au745uNU
node117 is the proxy node, with which I cluster all other 99 nodes (as
either disc or ram nodes). This seems to be the error (from the logs):
exception exit: {{badmatch,
{error,
{file_error,
"/var/lib/rabbitmq/mnesia/rabbit at node117
/msg_store_transient/file_summary.ets",
enoent}}},
Regards,
2011/3/28 Matthew Sackman <matthew at rabbitmq.com>
> Hi Advait,
>
> On Sun, Mar 27, 2011 at 11:01:53AM +0530, Advait Alai wrote:
> > I'm trying to add 150 nodes to a RabbitMQ cluster. After around 50 nodes,
> > the stop-reset-cluster-start iteration starts giving the error:
> >
> > Stopping node rabbit at node130 ...
> > ...done.
> > Resetting node rabbit at node130 ...
> > ...done.
> > Clustering node rabbit at node130 with [rabbit at node117] ...
> > ...done.
> > Starting node rabbit at node130 ...
> > Error: {cannot_start_application,rabbit,
> > {bad_return,
> > {{rabbit,start,[normal,[]]},
> > {'EXIT',{rabbit,failure_during_boot}}}}}
>
> Hmm, curious. I don't know why that happens. I've just successfully
> built a cluster of 100 nodes. I'm building from source, so using the
> Makefile. I did:
>
> 1. Edit the Makefile. Around line 153, remove the
> RABBITMQ_ALLOW_INPUT=true \
> line completely.
>
> 2. rabbitmq-server$ for t in $(seq 1 100); do make
> RABBITMQ_NODENAME=rabbit_$t RABBITMQ_NODE_PORT=$(echo "5672+$t"|bc)
> TMPDIR=/home/matthew/ssd/ cleandb run & done
>
> 3. rabbitmq-server/scripts$ for t in $(seq 2 100); do ./rabbitmqctl -n
> rabbit_$t at hazel stop_app ; ./rabbitmqctl -n rabbit_$t at hazel reset &&
> ./rabbitmqctl -n rabbit_$t at hazel cluster rabbit_1 at hazel rabbit_$t at hazel &&
> ./rabbitmqctl -n rabbit_$t at hazel start_app; done
>
> 4. Go make a cuppa tea.
>
> It takes a while, and slows down on a quadratic (which isn't surprising
> because we're forming an all-to-all mesh), and it eats up a lot of RAM.
> But it did get there for me:
>
> rabbitmq-server/scripts$ ./rabbitmqctl -n 'rabbit_1 at hazel' status
> Status of node rabbit_1 at hazel ...
> [{pid,7933},
> {running_applications,[{rabbit,"RabbitMQ","%%VSN%%"},
> {mnesia,"MNESIA CXC 138 12","4.4.17"},
> {os_mon,"CPO CXC 138 46","2.2.5"},
> {sasl,"SASL CXC 138 11","2.1.9.3"},
> {stdlib,"ERTS CXC 138 10","1.17.3"},
> {kernel,"ERTS CXC 138 10","2.14.3"}]},
> {nodes,[{disc,[rabbit_100 at hazel,rabbit_99 at hazel,rabbit_98 at hazel,
> rabbit_97 at hazel,rabbit_96 at hazel,rabbit_95 at hazel,
> rabbit_94 at hazel,rabbit_93 at hazel,rabbit_92 at hazel,
> rabbit_91 at hazel,rabbit_90 at hazel,rabbit_89 at hazel,
> rabbit_88 at hazel,rabbit_87 at hazel,rabbit_86 at hazel,
> rabbit_85 at hazel,rabbit_84 at hazel,rabbit_83 at hazel,
> rabbit_82 at hazel,rabbit_81 at hazel,rabbit_80 at hazel,
> rabbit_79 at hazel,rabbit_78 at hazel,rabbit_77 at hazel,
> rabbit_76 at hazel,rabbit_75 at hazel,rabbit_74 at hazel,
> rabbit_73 at hazel,rabbit_72 at hazel,rabbit_71 at hazel,
> rabbit_70 at hazel,rabbit_69 at hazel,rabbit_68 at hazel,
> rabbit_67 at hazel,rabbit_66 at hazel,rabbit_65 at hazel,
> rabbit_64 at hazel,rabbit_63 at hazel,rabbit_62 at hazel,
> rabbit_61 at hazel,rabbit_60 at hazel,rabbit_59 at hazel,
> rabbit_58 at hazel,rabbit_57 at hazel,rabbit_56 at hazel,
> rabbit_55 at hazel,rabbit_54 at hazel,rabbit_53 at hazel,
> rabbit_52 at hazel,rabbit_51 at hazel,rabbit_50 at hazel,
> rabbit_49 at hazel,rabbit_48 at hazel,rabbit_47 at hazel,
> rabbit_46 at hazel,rabbit_45 at hazel,rabbit_44 at hazel,
> rabbit_43 at hazel,rabbit_42 at hazel,rabbit_41 at hazel,
> rabbit_40 at hazel,rabbit_39 at hazel,rabbit_38 at hazel,
> rabbit_37 at hazel,rabbit_36 at hazel,rabbit_35 at hazel,
> rabbit_34 at hazel,rabbit_33 at hazel,rabbit_32 at hazel,
> rabbit_31 at hazel,rabbit_30 at hazel,rabbit_29 at hazel,
> rabbit_28 at hazel,rabbit_27 at hazel,rabbit_26 at hazel,
> rabbit_25 at hazel,rabbit_24 at hazel,rabbit_23 at hazel,
> rabbit_22 at hazel,rabbit_21 at hazel,rabbit_20 at hazel,
> rabbit_19 at hazel,rabbit_18 at hazel,rabbit_17 at hazel,
> rabbit_16 at hazel,rabbit_15 at hazel,rabbit_14 at hazel,
> rabbit_13 at hazel,rabbit_12 at hazel,rabbit_11 at hazel,
> rabbit_10 at hazel,rabbit_9 at hazel,rabbit_8 at hazel
> ,rabbit_7 at hazel,
> rabbit_6 at hazel,rabbit_5 at hazel,rabbit_4 at hazel,rabbit_3 at hazel
> ,
> rabbit_2 at hazel,rabbit_1 at hazel]}]},
> {running_nodes,[rabbit_100 at hazel,rabbit_99 at hazel,rabbit_98 at hazel,
> rabbit_97 at hazel,rabbit_96 at hazel,rabbit_95 at hazel,
> rabbit_94 at hazel,rabbit_93 at hazel,rabbit_92 at hazel,
> rabbit_91 at hazel,rabbit_90 at hazel,rabbit_89 at hazel,
> rabbit_88 at hazel,rabbit_87 at hazel,rabbit_86 at hazel,
> rabbit_85 at hazel,rabbit_84 at hazel,rabbit_83 at hazel,
> rabbit_82 at hazel,rabbit_81 at hazel,rabbit_80 at hazel,
> rabbit_79 at hazel,rabbit_78 at hazel,rabbit_77 at hazel,
> rabbit_76 at hazel,rabbit_75 at hazel,rabbit_74 at hazel,
> rabbit_73 at hazel,rabbit_72 at hazel,rabbit_71 at hazel,
> rabbit_70 at hazel,rabbit_69 at hazel,rabbit_68 at hazel,
> rabbit_67 at hazel,rabbit_66 at hazel,rabbit_65 at hazel,
> rabbit_64 at hazel,rabbit_63 at hazel,rabbit_62 at hazel,
> rabbit_61 at hazel,rabbit_60 at hazel,rabbit_59 at hazel,
> rabbit_58 at hazel,rabbit_57 at hazel,rabbit_56 at hazel,
> rabbit_55 at hazel,rabbit_54 at hazel,rabbit_53 at hazel,
> rabbit_52 at hazel,rabbit_51 at hazel,rabbit_50 at hazel,
> rabbit_49 at hazel,rabbit_48 at hazel,rabbit_47 at hazel,
> rabbit_46 at hazel,rabbit_45 at hazel,rabbit_44 at hazel,
> rabbit_43 at hazel,rabbit_42 at hazel,rabbit_41 at hazel,
> rabbit_40 at hazel,rabbit_39 at hazel,rabbit_38 at hazel,
> rabbit_37 at hazel,rabbit_36 at hazel,rabbit_35 at hazel,
> rabbit_34 at hazel,rabbit_33 at hazel,rabbit_32 at hazel,
> rabbit_31 at hazel,rabbit_30 at hazel,rabbit_29 at hazel,
> rabbit_28 at hazel,rabbit_27 at hazel,rabbit_26 at hazel,
> rabbit_25 at hazel,rabbit_24 at hazel,rabbit_23 at hazel,
> rabbit_22 at hazel,rabbit_21 at hazel,rabbit_20 at hazel,
> rabbit_19 at hazel,rabbit_18 at hazel,rabbit_17 at hazel,
> rabbit_16 at hazel,rabbit_15 at hazel,rabbit_14 at hazel,
> rabbit_13 at hazel,rabbit_12 at hazel,rabbit_11 at hazel,
> rabbit_10 at hazel,rabbit_9 at hazel,rabbit_8 at hazel
> ,rabbit_7 at hazel,
> rabbit_6 at hazel,rabbit_5 at hazel,rabbit_4 at hazel
> ,rabbit_3 at hazel,
> rabbit_2 at hazel,rabbit_1 at hazel]}]
> ...done.
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110403/e03f8b9d/attachment.htm>
More information about the rabbitmq-discuss
mailing list