Are there timelines on when such a mirrored queue implementation might make it out the door?<div><br clear="all">Mark Steele<br>Bering Media Inc.<br><div>+1 (416) 888-1009</div><br>
<br><br><div class="gmail_quote">On Fri, Jun 17, 2011 at 2:25 PM, Joseph Marlin <span dir="ltr"><<a href="mailto:joseph.a.marlin@gmail.com">joseph.a.marlin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
See the attachment on this message for details on high availability<br>
implementation coming soon that will change the ways queues on<br>
clusters work signficantly:<br>
<br>
<a href="http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-June/013304.html" target="_blank">http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-June/013304.html</a><br>
<br>
On Jun 17, 1:05�pm, Mark Steele <<a href="mailto:mste...@beringmedia.com">mste...@beringmedia.com</a>> wrote:<br>
> Hi Jon,<br>
><br>
> Rabbit will not scale to a large throughput of messages published to a<br>
> single queue as they all get delivered to a single node.<br>
><br>
> Here's one way to do it:<br>
><br>
> <a href="http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-March/01200." target="_blank">http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-March/01200.</a>..<br>
><br>
> The gist is:<br>
><br>
> � �- Rabbit doesn't scale on per queue basis. The answer: create many<br>
> � �queues, bound to the same exchange<br>
> � �- Use a direct exchange, which will rely on routing keys to decide which<br>
> � �queue to deliver a message to<br>
> � �- Pre-declare the queues on the nodes inside your cluster, bind them<br>
> � �using a known set of routing keys (eg: queueA with routing keys 1-100 on<br>
> � �server1, queueB with routing keys 101-200 on server2, etc...). Assuming<br>
> � �durable queues here...<br>
> � �- Have your publisher randomly pick a routing key and node to connect to<br>
> � �(or use a load balancer)<br>
> � �- Use the mandatory flag to ensure your message reaches a queue, if it<br>
> � �fails re-publish using a different randomly picked routing key<br>
> � �- Have your consumers connect to all queues<br>
><br>
> Using the technique described above, you can load balance incoming TCP<br>
> connections to a cluster of rabbit nodes. Load will get distributed based on<br>
> the randomly picked routing keys used by the publishers.<br>
><br>
> The problem with this is that HA is not addressed.<br>
><br>
> Mark Steele<br>
> Bering Media Inc.+1 (416) 888-1009<br>
><br>
> On Fri, Jun 17, 2011 at 11:51 AM, Jon Charette <<a href="mailto:jon.chare...@gmail.com">jon.chare...@gmail.com</a>>wrote:<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> > Greetings,<br>
><br>
> > We are investigating using RabbitMQ in a clustered environment to provide<br>
> > load balancing and some fault tolerance for an application currently under<br>
> > development.<br>
><br>
> > I'm not a developer, I'm the lead of our Operations team. �Clustering seems<br>
> > the best approach for my team as it simplifies our application configuration<br>
> > significantly.<br>
><br>
> > My question is this. �Our Dev team has told me that since clustering only<br>
> > provides node fault tolerance, it doesn't add any additional gain to our<br>
> > design. �Specifically, they mentioned that since a message queue only<br>
> > resides on one node,<br>
><br>
> > "All data/state required for the operation of a RabbitMQ broker is<br>
> > replicated across all nodes, for reliability and scaling, with full ACID<br>
> > properties. An exception to this are message queues, which currently only<br>
> > reside on the node that created them, though they are visible and reachable<br>
> > from all nodes. Future releases of RabbitMQ will introduce migration and<br>
> > replication of message queues."<br>
><br>
> > clustering doesn't allow us to scale based on the numbers of messages being<br>
> > sent to one queue. �Is this true, or does the cluster distribute messages<br>
> > sent to the same message queue throughout off of its member nodes? �My<br>
> > assumption based on the documentation is that this is the case, but if a<br>
> > node is lost, messages that were currently residing on that node are lost as<br>
> > well due to no fault tolerance.<br>
><br>
> > Thanks much.<br>
><br>
> > _______________________________________________<br>
> > rabbitmq-discuss mailing list<br>
> > <a href="mailto:rabbitmq-disc...@lists.rabbitmq.com">rabbitmq-disc...@lists.rabbitmq.com</a><br>
> ><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> rabbitmq-discuss mailing list<br>
> rabbitmq-disc...@lists.rabbitmq.comhttps://<a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</blockquote></div><br></div>