[rabbitmq-discuss] Clustering and data partitioning - how best to implement

Mon Sep 26 21:05:32 BST 2011

I'm trying to implement some of the ideas from this article:

http://blog.springsource.com/2011/04/01/routing-topologies-for-performance-and-scalability-with-rabbitmq/

Our situation is that we have two rabbit instances, not clustered, so
that we can spread out load. In our current setup, it's still possible
for us to hit one broker harder than the other, so ideally we'd like
to setup these nodes in a cluster. That way, we can have consumers
evenly distributed across brokers.

We could create one queue that all consumers consume from, but I'm not
sure that would solve our problems since that queue (and the broker it
was created on) would become a hotspot. Right?

So from that article, I like the idea of each node creating a queue,
e.g. myQueue.1, myQueue.2, etc. and having the producer round robin
them. What I don't know, is how the producer knows how many queues/
nodes there are. The producers could send to an exchange and just set
the routingKey, but still, I think the problem is the same.

I guess I could use the management plugin, but I'm trying to figure
out how to do that without issuing a request for every message I
produce.

I could imagine some kind of handshake when a producer comes online,
he sends a "who's-alive" message to a topic exchange and waits for
responses from consumers. Also, when a consumer comes online, he
immediately sends out a "I'm-alive" message. In a direct exchange
(either the default or one that I create), is a routingKey assumed to
be a queue name? If I made the queues auto_delete, what would happen
when the producer tries to send a message to a queue that no longer
exists? Would I get an exception such that I can catch it and send the
message to the next guy in the list?

Thanks,
Justin