[rabbitmq-discuss] Cluster with Single Message queue

[Mark Steele]

Hi Jon,

Rabbit will not scale to a large throughput of messages published to a
single queue as they all get delivered to a single node.

Here's one way to do it:

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-March/012007.html

The gist is:

   - Rabbit doesn't scale on per queue basis. The answer: create many
   queues, bound to the same exchange
   - Use a direct exchange, which will rely on routing keys to decide which
   queue to deliver a message to
   - Pre-declare the queues on the nodes inside your cluster, bind them
   using a known set of routing keys (eg: queueA with routing keys 1-100 on
   server1, queueB with routing keys 101-200 on server2, etc...). Assuming
   durable queues here...
   - Have your publisher randomly pick a routing key and node to connect to
   (or use a load balancer)
   - Use the mandatory flag to ensure your message reaches a queue, if it
   fails re-publish using a different randomly picked routing key
   - Have your consumers connect to all queues

Using the technique described above, you can load balance incoming TCP
connections to a cluster of rabbit nodes. Load will get distributed based on
the randomly picked routing keys used by the publishers.

The problem with this is that HA is not addressed.

Mark Steele
Bering Media Inc.
+1 (416) 888-1009



On Fri, Jun 17, 2011 at 11:51 AM, Jon Charette <jon.charette at gmail.com>wrote:

> Greetings,
>
>
>
> We are investigating using RabbitMQ in a clustered environment to provide
> load balancing and some fault tolerance for an application currently under
> development.
>
>
> I'm not a developer, I'm the lead of our Operations team.  Clustering seems
> the best approach for my team as it simplifies our application configuration
> significantly.
>
>
>
> My question is this.  Our Dev team has told me that since clustering only
> provides node fault tolerance, it doesn't add any additional gain to our
> design.  Specifically, they mentioned that since a message queue only
> resides on one node,
>
>
>
> "All data/state required for the operation of a RabbitMQ broker is
> replicated across all nodes, for reliability and scaling, with full ACID
> properties. An exception to this are message queues, which currently only
> reside on the node that created them, though they are visible and reachable
> from all nodes. Future releases of RabbitMQ will introduce migration and
> replication of message queues."
>
>
>
> clustering doesn't allow us to scale based on the numbers of messages being
> sent to one queue.  Is this true, or does the cluster distribute messages
> sent to the same message queue throughout off of its member nodes?  My
> assumption based on the documentation is that this is the case, but if a
> node is lost, messages that were currently residing on that node are lost as
> well due to no fault tolerance.
>
>
>
> Thanks much.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110617/95cce7b4/attachment-0001.htm>