[rabbitmq-discuss] Cluster with Single Message queue

Fri Jun 17 19:58:14 BST 2011

Are there timelines on when such a mirrored queue implementation might make
it out the door?

Mark Steele
Bering Media Inc.
+1 (416) 888-1009

On Fri, Jun 17, 2011 at 2:25 PM, Joseph Marlin <joseph.a.marlin at gmail.com>wrote:

> See the attachment on this message for details on high availability
> implementation coming soon that will change the ways queues on
> clusters work signficantly:
>
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-June/013304.html
>
> On Jun 17, 1:05 pm, Mark Steele <mste... at beringmedia.com> wrote:
> > Hi Jon,
> >
> > Rabbit will not scale to a large throughput of messages published to a
> > single queue as they all get delivered to a single node.
> >
> > Here's one way to do it:
> >
> > http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-March/01200...
> >
> > The gist is:
> >
> >    - Rabbit doesn't scale on per queue basis. The answer: create many
> >    queues, bound to the same exchange
> >    - Use a direct exchange, which will rely on routing keys to decide
> which
> >    queue to deliver a message to
> >    - Pre-declare the queues on the nodes inside your cluster, bind them
> >    using a known set of routing keys (eg: queueA with routing keys 1-100
> on
> >    server1, queueB with routing keys 101-200 on server2, etc...).
> Assuming
> >    durable queues here...
> >    - Have your publisher randomly pick a routing key and node to connect
> to
> >    (or use a load balancer)
> >    - Use the mandatory flag to ensure your message reaches a queue, if it
> >    fails re-publish using a different randomly picked routing key
> >    - Have your consumers connect to all queues
> >
> > Using the technique described above, you can load balance incoming TCP
> > connections to a cluster of rabbit nodes. Load will get distributed based
> on
> > the randomly picked routing keys used by the publishers.
> >
> > The problem with this is that HA is not addressed.
> >
> > Mark Steele
> > Bering Media Inc.+1 (416) 888-1009
> >
> > On Fri, Jun 17, 2011 at 11:51 AM, Jon Charette <jon.chare... at gmail.com
> >wrote:
> >
> >
> >
> >
> >
> >
> >
> > > Greetings,
> >
> > > We are investigating using RabbitMQ in a clustered environment to
> provide
> > > load balancing and some fault tolerance for an application currently
> under
> > > development.
> >
> > > I'm not a developer, I'm the lead of our Operations team.  Clustering
> seems
> > > the best approach for my team as it simplifies our application
> configuration
> > > significantly.
> >
> > > My question is this.  Our Dev team has told me that since clustering
> only
> > > provides node fault tolerance, it doesn't add any additional gain to
> our
> > > design.  Specifically, they mentioned that since a message queue only
> > > resides on one node,
> >
> > > "All data/state required for the operation of a RabbitMQ broker is
> > > replicated across all nodes, for reliability and scaling, with full
> ACID
> > > properties. An exception to this are message queues, which currently
> only
> > > reside on the node that created them, though they are visible and
> reachable
> > > from all nodes. Future releases of RabbitMQ will introduce migration
> and
> > > replication of message queues."
> >
> > > clustering doesn't allow us to scale based on the numbers of messages
> being
> > > sent to one queue.  Is this true, or does the cluster distribute
> messages
> > > sent to the same message queue throughout off of its member nodes?  My
> > > assumption based on the documentation is that this is the case, but if
> a
> > > node is lost, messages that were currently residing on that node are
> lost as
> > > well due to no fault tolerance.
> >
> > > Thanks much.
> >
> > > _______________________________________________
> > > rabbitmq-discuss mailing list
> > > rabbitmq-disc... at lists.rabbitmq.com
> > >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> >
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-disc... at lists.rabbitmq.comhttps://
> lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110617/a0471f9c/attachment-0001.htm>