[rabbitmq-discuss] Cluster size limit

Advait Alai advaitalai at gmail.com
Sun Apr 3 12:58:44 BST 2011


Hi Matthew,

Thanks a lot for the reply.

I am not sure why this error still appears after I attempt 'rabbitmqctl
start_app'. I tried disk clustering as well, with the same result. The error
logs look like this:

rabbit at node117.log:  http://pastebin.com/embhVaea
rabbit at node117.log.1: http://pastebin.com/Xa5ZnWrC
rabbit at node117-sasl.log: (empty)
rabbit at node117-sasl.log.1: http://pastebin.com/Au745uNU

node117 is the proxy node, with which I cluster all other 99 nodes (as
either disc or ram nodes). This seems to be the error (from the logs):

exception exit: {{badmatch,
                         {error,
                             {file_error,
                                 "/var/lib/rabbitmq/mnesia/rabbit at node117
/msg_store_transient/file_summary.ets",
                                 enoent}}},

Regards,

2011/3/28 Matthew Sackman <matthew at rabbitmq.com>

> Hi Advait,
>
> On Sun, Mar 27, 2011 at 11:01:53AM +0530, Advait Alai wrote:
> > I'm trying to add 150 nodes to a RabbitMQ cluster. After around 50 nodes,
> > the stop-reset-cluster-start iteration starts giving the error:
> >
> > Stopping node rabbit at node130 ...
> > ...done.
> > Resetting node rabbit at node130 ...
> > ...done.
> > Clustering node rabbit at node130 with [rabbit at node117] ...
> > ...done.
> > Starting node rabbit at node130 ...
> > Error: {cannot_start_application,rabbit,
> >            {bad_return,
> >                {{rabbit,start,[normal,[]]},
> >                 {'EXIT',{rabbit,failure_during_boot}}}}}
>
> Hmm, curious. I don't know why that happens. I've just successfully
> built a cluster of 100 nodes. I'm building from source, so using the
> Makefile. I did:
>
> 1. Edit the Makefile. Around line 153, remove the
>                RABBITMQ_ALLOW_INPUT=true \
>  line completely.
>
> 2. rabbitmq-server$ for t in $(seq 1 100); do make
> RABBITMQ_NODENAME=rabbit_$t RABBITMQ_NODE_PORT=$(echo "5672+$t"|bc)
> TMPDIR=/home/matthew/ssd/ cleandb run & done
>
> 3. rabbitmq-server/scripts$ for t in $(seq 2 100); do ./rabbitmqctl -n
> rabbit_$t at hazel stop_app ; ./rabbitmqctl -n rabbit_$t at hazel reset &&
> ./rabbitmqctl -n rabbit_$t at hazel cluster rabbit_1 at hazel rabbit_$t at hazel &&
> ./rabbitmqctl -n rabbit_$t at hazel start_app; done
>
> 4. Go make a cuppa tea.
>
> It takes a while, and slows down on a quadratic (which isn't surprising
> because we're forming an all-to-all mesh), and it eats up a lot of RAM.
> But it did get there for me:
>
> rabbitmq-server/scripts$ ./rabbitmqctl -n 'rabbit_1 at hazel' status
> Status of node rabbit_1 at hazel ...
> [{pid,7933},
>  {running_applications,[{rabbit,"RabbitMQ","%%VSN%%"},
>                        {mnesia,"MNESIA  CXC 138 12","4.4.17"},
>                        {os_mon,"CPO  CXC 138 46","2.2.5"},
>                        {sasl,"SASL  CXC 138 11","2.1.9.3"},
>                        {stdlib,"ERTS  CXC 138 10","1.17.3"},
>                        {kernel,"ERTS  CXC 138 10","2.14.3"}]},
>  {nodes,[{disc,[rabbit_100 at hazel,rabbit_99 at hazel,rabbit_98 at hazel,
>                rabbit_97 at hazel,rabbit_96 at hazel,rabbit_95 at hazel,
>                rabbit_94 at hazel,rabbit_93 at hazel,rabbit_92 at hazel,
>                rabbit_91 at hazel,rabbit_90 at hazel,rabbit_89 at hazel,
>                rabbit_88 at hazel,rabbit_87 at hazel,rabbit_86 at hazel,
>                rabbit_85 at hazel,rabbit_84 at hazel,rabbit_83 at hazel,
>                rabbit_82 at hazel,rabbit_81 at hazel,rabbit_80 at hazel,
>                rabbit_79 at hazel,rabbit_78 at hazel,rabbit_77 at hazel,
>                rabbit_76 at hazel,rabbit_75 at hazel,rabbit_74 at hazel,
>                rabbit_73 at hazel,rabbit_72 at hazel,rabbit_71 at hazel,
>                rabbit_70 at hazel,rabbit_69 at hazel,rabbit_68 at hazel,
>                rabbit_67 at hazel,rabbit_66 at hazel,rabbit_65 at hazel,
>                rabbit_64 at hazel,rabbit_63 at hazel,rabbit_62 at hazel,
>                rabbit_61 at hazel,rabbit_60 at hazel,rabbit_59 at hazel,
>                rabbit_58 at hazel,rabbit_57 at hazel,rabbit_56 at hazel,
>                rabbit_55 at hazel,rabbit_54 at hazel,rabbit_53 at hazel,
>                rabbit_52 at hazel,rabbit_51 at hazel,rabbit_50 at hazel,
>                rabbit_49 at hazel,rabbit_48 at hazel,rabbit_47 at hazel,
>                rabbit_46 at hazel,rabbit_45 at hazel,rabbit_44 at hazel,
>                rabbit_43 at hazel,rabbit_42 at hazel,rabbit_41 at hazel,
>                rabbit_40 at hazel,rabbit_39 at hazel,rabbit_38 at hazel,
>                rabbit_37 at hazel,rabbit_36 at hazel,rabbit_35 at hazel,
>                rabbit_34 at hazel,rabbit_33 at hazel,rabbit_32 at hazel,
>                rabbit_31 at hazel,rabbit_30 at hazel,rabbit_29 at hazel,
>                rabbit_28 at hazel,rabbit_27 at hazel,rabbit_26 at hazel,
>                rabbit_25 at hazel,rabbit_24 at hazel,rabbit_23 at hazel,
>                rabbit_22 at hazel,rabbit_21 at hazel,rabbit_20 at hazel,
>                rabbit_19 at hazel,rabbit_18 at hazel,rabbit_17 at hazel,
>                rabbit_16 at hazel,rabbit_15 at hazel,rabbit_14 at hazel,
>                rabbit_13 at hazel,rabbit_12 at hazel,rabbit_11 at hazel,
>                rabbit_10 at hazel,rabbit_9 at hazel,rabbit_8 at hazel
> ,rabbit_7 at hazel,
>                rabbit_6 at hazel,rabbit_5 at hazel,rabbit_4 at hazel,rabbit_3 at hazel
> ,
>                rabbit_2 at hazel,rabbit_1 at hazel]}]},
>  {running_nodes,[rabbit_100 at hazel,rabbit_99 at hazel,rabbit_98 at hazel,
>                 rabbit_97 at hazel,rabbit_96 at hazel,rabbit_95 at hazel,
>                 rabbit_94 at hazel,rabbit_93 at hazel,rabbit_92 at hazel,
>                 rabbit_91 at hazel,rabbit_90 at hazel,rabbit_89 at hazel,
>                 rabbit_88 at hazel,rabbit_87 at hazel,rabbit_86 at hazel,
>                 rabbit_85 at hazel,rabbit_84 at hazel,rabbit_83 at hazel,
>                 rabbit_82 at hazel,rabbit_81 at hazel,rabbit_80 at hazel,
>                 rabbit_79 at hazel,rabbit_78 at hazel,rabbit_77 at hazel,
>                 rabbit_76 at hazel,rabbit_75 at hazel,rabbit_74 at hazel,
>                 rabbit_73 at hazel,rabbit_72 at hazel,rabbit_71 at hazel,
>                 rabbit_70 at hazel,rabbit_69 at hazel,rabbit_68 at hazel,
>                 rabbit_67 at hazel,rabbit_66 at hazel,rabbit_65 at hazel,
>                 rabbit_64 at hazel,rabbit_63 at hazel,rabbit_62 at hazel,
>                 rabbit_61 at hazel,rabbit_60 at hazel,rabbit_59 at hazel,
>                 rabbit_58 at hazel,rabbit_57 at hazel,rabbit_56 at hazel,
>                 rabbit_55 at hazel,rabbit_54 at hazel,rabbit_53 at hazel,
>                 rabbit_52 at hazel,rabbit_51 at hazel,rabbit_50 at hazel,
>                 rabbit_49 at hazel,rabbit_48 at hazel,rabbit_47 at hazel,
>                 rabbit_46 at hazel,rabbit_45 at hazel,rabbit_44 at hazel,
>                 rabbit_43 at hazel,rabbit_42 at hazel,rabbit_41 at hazel,
>                 rabbit_40 at hazel,rabbit_39 at hazel,rabbit_38 at hazel,
>                 rabbit_37 at hazel,rabbit_36 at hazel,rabbit_35 at hazel,
>                 rabbit_34 at hazel,rabbit_33 at hazel,rabbit_32 at hazel,
>                 rabbit_31 at hazel,rabbit_30 at hazel,rabbit_29 at hazel,
>                 rabbit_28 at hazel,rabbit_27 at hazel,rabbit_26 at hazel,
>                 rabbit_25 at hazel,rabbit_24 at hazel,rabbit_23 at hazel,
>                 rabbit_22 at hazel,rabbit_21 at hazel,rabbit_20 at hazel,
>                 rabbit_19 at hazel,rabbit_18 at hazel,rabbit_17 at hazel,
>                 rabbit_16 at hazel,rabbit_15 at hazel,rabbit_14 at hazel,
>                 rabbit_13 at hazel,rabbit_12 at hazel,rabbit_11 at hazel,
>                 rabbit_10 at hazel,rabbit_9 at hazel,rabbit_8 at hazel
> ,rabbit_7 at hazel,
>                 rabbit_6 at hazel,rabbit_5 at hazel,rabbit_4 at hazel
> ,rabbit_3 at hazel,
>                 rabbit_2 at hazel,rabbit_1 at hazel]}]
> ...done.
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110403/e03f8b9d/attachment.htm>


More information about the rabbitmq-discuss mailing list