[rabbitmq-discuss] Cluster size limit

[Matthew Sackman]

Hi Advait,

On Sun, Mar 27, 2011 at 11:01:53AM +0530, Advait Alai wrote:
> I'm trying to add 150 nodes to a RabbitMQ cluster. After around 50 nodes,
> the stop-reset-cluster-start iteration starts giving the error:
> 
> Stopping node rabbit at node130 ...
> ...done.
> Resetting node rabbit at node130 ...
> ...done.
> Clustering node rabbit at node130 with [rabbit at node117] ...
> ...done.
> Starting node rabbit at node130 ...
> Error: {cannot_start_application,rabbit,
>            {bad_return,
>                {{rabbit,start,[normal,[]]},
>                 {'EXIT',{rabbit,failure_during_boot}}}}}

Hmm, curious. I don't know why that happens. I've just successfully
built a cluster of 100 nodes. I'm building from source, so using the
Makefile. I did:

1. Edit the Makefile. Around line 153, remove the
		RABBITMQ_ALLOW_INPUT=true \
  line completely.

2. rabbitmq-server$ for t in $(seq 1 100); do make RABBITMQ_NODENAME=rabbit_$t RABBITMQ_NODE_PORT=$(echo "5672+$t"|bc) TMPDIR=/home/matthew/ssd/ cleandb run & done

3. rabbitmq-server/scripts$ for t in $(seq 2 100); do ./rabbitmqctl -n rabbit_$t at hazel stop_app ; ./rabbitmqctl -n rabbit_$t at hazel reset && ./rabbitmqctl -n rabbit_$t at hazel cluster rabbit_1 at hazel rabbit_$t at hazel && ./rabbitmqctl -n rabbit_$t at hazel start_app; done

4. Go make a cuppa tea.

It takes a while, and slows down on a quadratic (which isn't surprising
because we're forming an all-to-all mesh), and it eats up a lot of RAM.
But it did get there for me:

rabbitmq-server/scripts$ ./rabbitmqctl -n 'rabbit_1 at hazel' status
Status of node rabbit_1 at hazel ...
[{pid,7933},
 {running_applications,[{rabbit,"RabbitMQ","%%VSN%%"},
                        {mnesia,"MNESIA  CXC 138 12","4.4.17"},
                        {os_mon,"CPO  CXC 138 46","2.2.5"},
                        {sasl,"SASL  CXC 138 11","2.1.9.3"},
                        {stdlib,"ERTS  CXC 138 10","1.17.3"},
                        {kernel,"ERTS  CXC 138 10","2.14.3"}]},
 {nodes,[{disc,[rabbit_100 at hazel,rabbit_99 at hazel,rabbit_98 at hazel,
                rabbit_97 at hazel,rabbit_96 at hazel,rabbit_95 at hazel,
                rabbit_94 at hazel,rabbit_93 at hazel,rabbit_92 at hazel,
                rabbit_91 at hazel,rabbit_90 at hazel,rabbit_89 at hazel,
                rabbit_88 at hazel,rabbit_87 at hazel,rabbit_86 at hazel,
                rabbit_85 at hazel,rabbit_84 at hazel,rabbit_83 at hazel,
                rabbit_82 at hazel,rabbit_81 at hazel,rabbit_80 at hazel,
                rabbit_79 at hazel,rabbit_78 at hazel,rabbit_77 at hazel,
                rabbit_76 at hazel,rabbit_75 at hazel,rabbit_74 at hazel,
                rabbit_73 at hazel,rabbit_72 at hazel,rabbit_71 at hazel,
                rabbit_70 at hazel,rabbit_69 at hazel,rabbit_68 at hazel,
                rabbit_67 at hazel,rabbit_66 at hazel,rabbit_65 at hazel,
                rabbit_64 at hazel,rabbit_63 at hazel,rabbit_62 at hazel,
                rabbit_61 at hazel,rabbit_60 at hazel,rabbit_59 at hazel,
                rabbit_58 at hazel,rabbit_57 at hazel,rabbit_56 at hazel,
                rabbit_55 at hazel,rabbit_54 at hazel,rabbit_53 at hazel,
                rabbit_52 at hazel,rabbit_51 at hazel,rabbit_50 at hazel,
                rabbit_49 at hazel,rabbit_48 at hazel,rabbit_47 at hazel,
                rabbit_46 at hazel,rabbit_45 at hazel,rabbit_44 at hazel,
                rabbit_43 at hazel,rabbit_42 at hazel,rabbit_41 at hazel,
                rabbit_40 at hazel,rabbit_39 at hazel,rabbit_38 at hazel,
                rabbit_37 at hazel,rabbit_36 at hazel,rabbit_35 at hazel,
                rabbit_34 at hazel,rabbit_33 at hazel,rabbit_32 at hazel,
                rabbit_31 at hazel,rabbit_30 at hazel,rabbit_29 at hazel,
                rabbit_28 at hazel,rabbit_27 at hazel,rabbit_26 at hazel,
                rabbit_25 at hazel,rabbit_24 at hazel,rabbit_23 at hazel,
                rabbit_22 at hazel,rabbit_21 at hazel,rabbit_20 at hazel,
                rabbit_19 at hazel,rabbit_18 at hazel,rabbit_17 at hazel,
                rabbit_16 at hazel,rabbit_15 at hazel,rabbit_14 at hazel,
                rabbit_13 at hazel,rabbit_12 at hazel,rabbit_11 at hazel,
                rabbit_10 at hazel,rabbit_9 at hazel,rabbit_8 at hazel,rabbit_7 at hazel,
                rabbit_6 at hazel,rabbit_5 at hazel,rabbit_4 at hazel,rabbit_3 at hazel,
                rabbit_2 at hazel,rabbit_1 at hazel]}]},
 {running_nodes,[rabbit_100 at hazel,rabbit_99 at hazel,rabbit_98 at hazel,
                 rabbit_97 at hazel,rabbit_96 at hazel,rabbit_95 at hazel,
                 rabbit_94 at hazel,rabbit_93 at hazel,rabbit_92 at hazel,
                 rabbit_91 at hazel,rabbit_90 at hazel,rabbit_89 at hazel,
                 rabbit_88 at hazel,rabbit_87 at hazel,rabbit_86 at hazel,
                 rabbit_85 at hazel,rabbit_84 at hazel,rabbit_83 at hazel,
                 rabbit_82 at hazel,rabbit_81 at hazel,rabbit_80 at hazel,
                 rabbit_79 at hazel,rabbit_78 at hazel,rabbit_77 at hazel,
                 rabbit_76 at hazel,rabbit_75 at hazel,rabbit_74 at hazel,
                 rabbit_73 at hazel,rabbit_72 at hazel,rabbit_71 at hazel,
                 rabbit_70 at hazel,rabbit_69 at hazel,rabbit_68 at hazel,
                 rabbit_67 at hazel,rabbit_66 at hazel,rabbit_65 at hazel,
                 rabbit_64 at hazel,rabbit_63 at hazel,rabbit_62 at hazel,
                 rabbit_61 at hazel,rabbit_60 at hazel,rabbit_59 at hazel,
                 rabbit_58 at hazel,rabbit_57 at hazel,rabbit_56 at hazel,
                 rabbit_55 at hazel,rabbit_54 at hazel,rabbit_53 at hazel,
                 rabbit_52 at hazel,rabbit_51 at hazel,rabbit_50 at hazel,
                 rabbit_49 at hazel,rabbit_48 at hazel,rabbit_47 at hazel,
                 rabbit_46 at hazel,rabbit_45 at hazel,rabbit_44 at hazel,
                 rabbit_43 at hazel,rabbit_42 at hazel,rabbit_41 at hazel,
                 rabbit_40 at hazel,rabbit_39 at hazel,rabbit_38 at hazel,
                 rabbit_37 at hazel,rabbit_36 at hazel,rabbit_35 at hazel,
                 rabbit_34 at hazel,rabbit_33 at hazel,rabbit_32 at hazel,
                 rabbit_31 at hazel,rabbit_30 at hazel,rabbit_29 at hazel,
                 rabbit_28 at hazel,rabbit_27 at hazel,rabbit_26 at hazel,
                 rabbit_25 at hazel,rabbit_24 at hazel,rabbit_23 at hazel,
                 rabbit_22 at hazel,rabbit_21 at hazel,rabbit_20 at hazel,
                 rabbit_19 at hazel,rabbit_18 at hazel,rabbit_17 at hazel,
                 rabbit_16 at hazel,rabbit_15 at hazel,rabbit_14 at hazel,
                 rabbit_13 at hazel,rabbit_12 at hazel,rabbit_11 at hazel,
                 rabbit_10 at hazel,rabbit_9 at hazel,rabbit_8 at hazel,rabbit_7 at hazel,
                 rabbit_6 at hazel,rabbit_5 at hazel,rabbit_4 at hazel,rabbit_3 at hazel,
                 rabbit_2 at hazel,rabbit_1 at hazel]}]
...done.

Matthew