[rabbitmq-discuss] Clustering and data partitioning - how best to implement

Helena Edelson hedelson at vmware.com
Tue Sep 27 01:06:14 BST 2011


Hi Justin,

Luckily, things are much simpler than what your questions lead to :-)

I think you're looking for is the round robin/competing consumers. I had a typo on that blog post which is now fixed that may have caused confusion.
Queue -> multiple consumers  http://www.rabbitmq.com/tutorials/tutorial-two-python.html
Your producer simply publishes to exchanges. The RabbitMQ server knows where to route the messages (via bindings between queues and exchanges), the Producer is decoupled completely from any endpoint awareness. This decoupling is key.
 
Do look at http://www.rabbitmq.com/ha.html also talks about queues in a cluster. It will answer a lot of your questions. 

You definitely want more than one queue...particularly if you are in any kind of scalable heavy load situation. Generally you have many queues, often partitioned logically. Do take note the performance issues to watch for in the post.

I hope that helps a bit, 
Helena  
________________________________________
From: alexis.richardson at gmail.com [alexis.richardson at gmail.com] On Behalf Of Alexis Richardson [alexis at rabbitmq.com]
Sent: Monday, September 26, 2011 2:10 PM
To: justin.adam.miller at gmail.com
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Clustering and data partitioning - how best to implement

Does this help?

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-September/015263.html




On Mon, Sep 26, 2011 at 9:05 PM, justin.adam.miller at gmail.com
<justin.adam.miller at gmail.com> wrote:
> I'm trying to implement some of the ideas from this article:
>
> http://blog.springsource.com/2011/04/01/routing-topologies-for-performance-and-scalability-with-rabbitmq/
>
> Our situation is that we have two rabbit instances, not clustered, so
> that we can spread out load. In our current setup, it's still possible
> for us to hit one broker harder than the other, so ideally we'd like
> to setup these nodes in a cluster. That way, we can have consumers
> evenly distributed across brokers.
>
> We could create one queue that all consumers consume from, but I'm not
> sure that would solve our problems since that queue (and the broker it
> was created on) would become a hotspot. Right?
>
> So from that article, I like the idea of each node creating a queue,
> e.g. myQueue.1, myQueue.2, etc. and having the producer round robin
> them. What I don't know, is how the producer knows how many queues/
> nodes there are. The producers could send to an exchange and just set
> the routingKey, but still, I think the problem is the same.
>
> I guess I could use the management plugin, but I'm trying to figure
> out how to do that without issuing a request for every message I
> produce.
>
> I could imagine some kind of handshake when a producer comes online,
> he sends a "who's-alive" message to a topic exchange and waits for
> responses from consumers. Also, when a consumer comes online, he
> immediately sends out a "I'm-alive" message. In a direct exchange
> (either the default or one that I create), is a routingKey assumed to
> be a queue name? If I made the queues auto_delete, what would happen
> when the producer tries to send a message to a queue that no longer
> exists? Would I get an exception such that I can catch it and send the
> message to the next guy in the list?
>
> Thanks,
> Justin
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>


More information about the rabbitmq-discuss mailing list