[rabbitmq-discuss] Cluster size limit

Sun Mar 27 22:25:59 BST 2011

Advait

There is no benefit to clustering terminal nodes which (as I
understand it) are not sharing internal state with each other.
Instead look to share queues and exchanges by running them on brokers,
and connecting multiple terminal nodes to those brokers.

You should run RabbitMQ clients at terminal nodes.  Connect them to
brokers running at non-terminal nodes.  For broker-broker links, use
Shovel.  We are working on a federation model that is more flexible
than shovel - if you are interested, talk to Simon (cc'd).

Does this help?

alexis

On Sun, Mar 27, 2011 at 9:33 PM, Advait Alai <advaitalai at gmail.com> wrote:
>> However - RabbitMQ clustering is not designed as a way to create
>> pubsub overlays for the wide area.  Its primary goal is scalability of
>> one broker by adding nodes on the same LAN.  This is for, eg, cases
>> where the number of subscriptions (or "bindings") on the broker grows
>> beyond what one machine can physically cope with.
>>
>> Now, this does not stop you wiring up 1,000s of RabbitMQ brokers.  But
>> using RabbitMQ clustering is the wrong way to do that.  Can you tell
>> us a bit more about the pubsub topology you want to study, please?
>> That may help us suggest the optimal way to solve your problem..
>
> The topology looks like this: http://imgur.com/8iep0 (taken
> from http://ctieware.eng.monash.edu.au/twiki/bin/view/Simulation/LargePacket-switchingNetworkTopologies)
> The terminal nodes shown in the topology are either publishers or
> subscribers, and I want RabbitMQ to run as a middleware over the entire
> network. The network (routers and links) are simulated by NS3, So it is
> essentially a set of point to point links, where each link is a separate
> network. The terminal nodes are VMs (that would run the amqp/rabbitmq
> scripts) which I am attempting to cluster so they can use a common set of
> queues and exchanges. So all terminal nodes are in different LANs.
>>
>> Also, can you explain why QoS studies require the study of 1,000s of
>> nodes?  Perhaps you mean 1,000s of clients?
>
> I'm not sure I understood -- I'm trying to provide high fidelity results, so
> I thought the testbed should be atleast of the same order of size as the
> network on which it needs to be deployed (>50000 nodes). There are a few
> strict IEEE standards regarding this network that specify the minimum QoS
> guarantees required.
> Is there an easier way to do this?
> Thanks
>>
>> On Sun, Mar 27, 2011 at 7:11 PM, Advait Alai <advaitalai at gmail.com> wrote:
>> > Thanks for the reply --
>> > On Sun, Mar 27, 2011 at 9:38 PM, Jerry Kuch <jerryk at vmware.com> wrote:
>> >>
>> >> 150 is a pretty big sounding cluster...  Out of curiosity, what's
>> >> motivating you to go so big (if you don't mind saying)?
>> >
>> > I am doing a QoS analysis of publish-subscribe overlays using RabbitMQ
>> > in
>> > wide area networks (These are country wide networks, so even a 1000
>> > nodes
>> > might be insufficient :-)) So stuff like packet delay, loss,
>> > out-of-order
>> > delivery etc.
>> >
>> >>
>> >> On that note, because RabbitMQ clustering is based on Erlang
>> >> distribution,
>> >> the current practical limit you'll probably run up against is somewhat
>> >> lower
>> >> than the 150 you have in mind.  Something more like 32 to 64.
>> >
>> > Is there a configuration that would let me scale to >64 nodes, even if
>> > it
>> > would not be practical? And in case Erlang does not scale well, I'll
>> > probably have to resort to entirely another middleware -- any
>> > suggestions
>> > that would work on a larger number of nodes?
>> >
>> >>
>> >> If you can say more about your goals it's likely that someone on the
>> >> Rabbit team can suggest something helpful.
>> >
>> > The analysis I'm carrying out on >100 nodes is actually on a single
>> > system.
>> > These 'nodes' are actually many light weight linux containers (more or
>> > less
>> > virtual machines) connected by a simulated NS3 network topology. But I
>> > doubt
>> > this would be the cause of the clustering problem, as <50 nodes were
>> > clustering without any difficulty.
>> >
>> >>
>> >> On Mar 26, 2011, at 10:32 PM, "Advait Alai" <advaitalai at gmail.com>
>> >> wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > I'm trying to add 150 nodes to a RabbitMQ cluster. After around 50
>> >> > nodes, the stop-reset-cluster-start iteration starts giving the
>> >> > error:
>> >> >
>> >> > Stopping node rabbit at node130 ...
>> >> > ...done.
>> >> > Resetting node rabbit at node130 ...
>> >> > ...done.
>> >> > Clustering node rabbit at node130 with [rabbit at node117] ...
>> >> > ...done.
>> >> > Starting node rabbit at node130 ...
>> >> > Error: {cannot_start_application,rabbit,
>> >> >            {bad_return,
>> >> >                {{rabbit,start,[normal,[]]},
>> >> >                 {'EXIT',{rabbit,failure_during_boot}}}}}
>> >> >
>> >> > Note that I am sequentially adding nodes to build a cluster (as an
>> >> > initialization step) before creating any queues/exchanges or running
>> >> > any
>> >> > amqp script.
>> >> >
>> >> > How do I solve this problem? Is it because RabbitMQ imposes a hard
>> >> > cluster size limit?
>> >> >
>> >> > Also, does RabbitMQ scale well to around 1000 nodes?
>> >> >
>> >> > Thanks
>> >> > _______________________________________________
>> >> > rabbitmq-discuss mailing list
>> >> > rabbitmq-discuss at lists.rabbitmq.com
>> >> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >
>> >
>> > _______________________________________________
>> > rabbitmq-discuss mailing list
>> > rabbitmq-discuss at lists.rabbitmq.com
>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >
>> >
>
>