<div>Alexis, </div><div><br></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8">Thanks for the inputs. </div><br><div class="gmail_quote">On Mon, Mar 28, 2011 at 2:55 AM, Alexis Richardson <span dir="ltr"><<a href="mailto:alexis@rabbitmq.com">alexis@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Advait<br>
<br>
There is no benefit to clustering terminal nodes which (as I<br>
understand it) are not sharing internal state with each other.<br>
Instead look to share queues and exchanges by running them on brokers,<br>
and connecting multiple terminal nodes to those brokers.<br></blockquote><div><br></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div>I am running the entire setup -- VMs and the network on a single system. <meta http-equiv="content-type" content="text/html; charset=utf-8">Does that count as sharing the same internal state? The VMs mount the same bare file system, so, for example, I could skip replicating erlang-cookies before clustering (/var/lib/rabbitmq is common). Of course, they have virtualized/individual network stacks and interfaces.</div>
</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
You should run RabbitMQ clients at terminal nodes. Connect them to<br>
brokers running at non-terminal nodes. For broker-broker links, use<br>
Shovel. We are working on a federation model that is more flexible<br>
than shovel - if you are interested, talk to Simon (cc'd).<br></blockquote><div><br></div><div>Ah, this might be a problem, as I cannot run any third party applications on non terminal nodes. These nodes are primitives of NS-3 -- just objects responsible for routing etc. Since I did not have any control on non terminal nodes, I had to cluster terminal nodes and assumed that the middleware would take care of setting up overlay paths from publishers to subscribers. My bad, should've understood how clustering worked before starting :-/</div>
<div><br></div><div>But coming back to the errors cropping up, is there a way to increase the cluster limit size, even if there is no added benefit of clustering this way?</div><div><br></div><div>@Simon -- could you please direct me to any documentation for the federation model I could use here? Thanks</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div><div class="h5">
On Sun, Mar 27, 2011 at 9:33 PM, Advait Alai <<a href="mailto:advaitalai@gmail.com">advaitalai@gmail.com</a>> wrote:<br>
>> However - RabbitMQ clustering is not designed as a way to create<br>
>> pubsub overlays for the wide area. Its primary goal is scalability of<br>
>> one broker by adding nodes on the same LAN. This is for, eg, cases<br>
>> where the number of subscriptions (or "bindings") on the broker grows<br>
>> beyond what one machine can physically cope with.<br>
>><br>
>> Now, this does not stop you wiring up 1,000s of RabbitMQ brokers. But<br>
>> using RabbitMQ clustering is the wrong way to do that. Can you tell<br>
>> us a bit more about the pubsub topology you want to study, please?<br>
>> That may help us suggest the optimal way to solve your problem..<br>
><br>
> The topology looks like this: <a href="http://imgur.com/8iep0" target="_blank">http://imgur.com/8iep0</a> (taken<br>
> from <a href="http://ctieware.eng.monash.edu.au/twiki/bin/view/Simulation/LargePacket-switchingNetworkTopologies" target="_blank">http://ctieware.eng.monash.edu.au/twiki/bin/view/Simulation/LargePacket-switchingNetworkTopologies</a>)<br>
> The terminal nodes shown in the topology are either publishers or<br>
> subscribers, and I want RabbitMQ to run as a middleware over the entire<br>
> network. The network (routers and links) are simulated by NS3, So it is<br>
> essentially a set of point to point links, where each link is a separate<br>
> network. The terminal nodes are VMs (that would run the amqp/rabbitmq<br>
> scripts) which I am attempting to cluster so they can use a common set of<br>
> queues and exchanges. So all terminal nodes are in different LANs.<br>
>><br>
>> Also, can you explain why QoS studies require the study of 1,000s of<br>
>> nodes? Perhaps you mean 1,000s of clients?<br>
><br>
> I'm not sure I understood -- I'm trying to provide high fidelity results, so<br>
> I thought the testbed should be atleast of the same order of size as the<br>
> network on which it needs to be deployed (>50000 nodes). There are a few<br>
> strict IEEE standards regarding this network that specify the minimum QoS<br>
> guarantees required.<br>
> Is there an easier way to do this?<br>
> Thanks<br>
>><br>
>> On Sun, Mar 27, 2011 at 7:11 PM, Advait Alai <<a href="mailto:advaitalai@gmail.com">advaitalai@gmail.com</a>> wrote:<br>
>> > Thanks for the reply --<br>
>> > On Sun, Mar 27, 2011 at 9:38 PM, Jerry Kuch <<a href="mailto:jerryk@vmware.com">jerryk@vmware.com</a>> wrote:<br>
>> >><br>
>> >> 150 is a pretty big sounding cluster... Out of curiosity, what's<br>
>> >> motivating you to go so big (if you don't mind saying)?<br>
>> ><br>
>> > I am doing a QoS analysis of publish-subscribe overlays using RabbitMQ<br>
>> > in<br>
>> > wide area networks (These are country wide networks, so even a 1000<br>
>> > nodes<br>
>> > might be insufficient :-)) So stuff like packet delay, loss,<br>
>> > out-of-order<br>
>> > delivery etc.<br>
>> ><br>
>> >><br>
>> >> On that note, because RabbitMQ clustering is based on Erlang<br>
>> >> distribution,<br>
>> >> the current practical limit you'll probably run up against is somewhat<br>
>> >> lower<br>
>> >> than the 150 you have in mind. Something more like 32 to 64.<br>
>> ><br>
>> > Is there a configuration that would let me scale to >64 nodes, even if<br>
>> > it<br>
>> > would not be practical? And in case Erlang does not scale well, I'll<br>
>> > probably have to resort to entirely another middleware -- any<br>
>> > suggestions<br>
>> > that would work on a larger number of nodes?<br>
>> ><br>
>> >><br>
>> >> If you can say more about your goals it's likely that someone on the<br>
>> >> Rabbit team can suggest something helpful.<br>
>> ><br>
>> > The analysis I'm carrying out on >100 nodes is actually on a single<br>
>> > system.<br>
>> > These 'nodes' are actually many light weight linux containers (more or<br>
>> > less<br>
>> > virtual machines) connected by a simulated NS3 network topology. But I<br>
>> > doubt<br>
>> > this would be the cause of the clustering problem, as <50 nodes were<br>
>> > clustering without any difficulty.<br>
>> ><br>
>> >><br>
>> >> On Mar 26, 2011, at 10:32 PM, "Advait Alai" <<a href="mailto:advaitalai@gmail.com">advaitalai@gmail.com</a>><br>
>> >> wrote:<br>
>> >><br>
>> >> > Hi,<br>
>> >> ><br>
>> >> > I'm trying to add 150 nodes to a RabbitMQ cluster. After around 50<br>
>> >> > nodes, the stop-reset-cluster-start iteration starts giving the<br>
>> >> > error:<br>
>> >> ><br>
>> >> > Stopping node rabbit@node130 ...<br>
>> >> > ...done.<br>
>> >> > Resetting node rabbit@node130 ...<br>
>> >> > ...done.<br>
>> >> > Clustering node rabbit@node130 with [rabbit@node117] ...<br>
>> >> > ...done.<br>
>> >> > Starting node rabbit@node130 ...<br>
>> >> > Error: {cannot_start_application,rabbit,<br>
>> >> > {bad_return,<br>
>> >> > {{rabbit,start,[normal,[]]},<br>
>> >> > {'EXIT',{rabbit,failure_during_boot}}}}}<br>
>> >> ><br>
>> >> > Note that I am sequentially adding nodes to build a cluster (as an<br>
>> >> > initialization step) before creating any queues/exchanges or running<br>
>> >> > any<br>
>> >> > amqp script.<br>
>> >> ><br>
>> >> > How do I solve this problem? Is it because RabbitMQ imposes a hard<br>
>> >> > cluster size limit?<br>
>> >> ><br>
>> >> > Also, does RabbitMQ scale well to around 1000 nodes?<br>
>> >> ><br>
>> >> > Thanks<br>
>> >> > _______________________________________________<br>
>> >> > rabbitmq-discuss mailing list<br>
>> >> > <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
>> >> > <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > rabbitmq-discuss mailing list<br>
>> > <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
>> > <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
>> ><br>
>> ><br>
><br>
><br>
</div></div></blockquote></div><br>