[rabbitmq-discuss] Clustering and data partitioning - how best to implement

[Helena Edelson]

Jason, you mean a topic exchange? If so, yes topic exchanges are great to distribute your traffic from the one queue with high load to n-queues, each queue bound with a pattern matching a 'sub topic' if you will. That and set up wiretaps, etc.
 
Helena
________________________________________
From: Justin Miller [justin.adam.miller at gmail.com]
Sent: Monday, September 26, 2011 6:46 PM
To: Helena Edelson
Cc: rabbitmq-discuss at lists.rabbitmq.com; Alexis Richardson
Subject: Re: [rabbitmq-discuss] Clustering and data partitioning - how best to implement

Helena, all,

Thanks for the responses. Actually the hashing exchange is exactly
what I'm looking for.

I already rely on the round robin behavior of queues, but our problem
is that we can't logically partition our data any further and we have
one queue that gets more requests than the others.

Right now, we manually break up the queue into those subqueues, but
that's not the most maintainable solution nor does it address the
problem in a cluster.

Given our limitations, the hashing exchange seems like the best solution, no?

Thanks
Justin

On Sep 26, 2011, at 8:06 PM, Helena Edelson <hedelson at vmware.com> wrote:

> Hi Justin,
>
> Luckily, things are much simpler than what your questions lead to :-)
>
> I think you're looking for is the round robin/competing consumers. I had a typo on that blog post which is now fixed that may have caused confusion.
> Queue -> multiple consumers  http://www.rabbitmq.com/tutorials/tutorial-two-python.html
> Your producer simply publishes to exchanges. The RabbitMQ server knows where to route the messages (via bindings between queues and exchanges), the Producer is decoupled completely from any endpoint awareness. This decoupling is key.
>
> Do look at http://www.rabbitmq.com/ha.html also talks about queues in a cluster. It will answer a lot of your questions.
>
> You definitely want more than one queue...particularly if you are in any kind of scalable heavy load situation. Generally you have many queues, often partitioned logically. Do take note the performance issues to watch for in the post.
>
> I hope that helps a bit,
> Helena
> ________________________________________
> From: alexis.richardson at gmail.com [alexis.richardson at gmail.com] On Behalf Of Alexis Richardson [alexis at rabbitmq.com]
> Sent: Monday, September 26, 2011 2:10 PM
> To: justin.adam.miller at gmail.com
> Cc: rabbitmq-discuss at lists.rabbitmq.com
> Subject: Re: [rabbitmq-discuss] Clustering and data partitioning - how best to implement
>
> Does this help?
>
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-September/015263.html
>
>
>
>
> On Mon, Sep 26, 2011 at 9:05 PM, justin.adam.miller at gmail.com
> <justin.adam.miller at gmail.com> wrote:
>> I'm trying to implement some of the ideas from this article:
>>
>> http://blog.springsource.com/2011/04/01/routing-topologies-for-performance-and-scalability-with-rabbitmq/
>>
>> Our situation is that we have two rabbit instances, not clustered, so
>> that we can spread out load. In our current setup, it's still possible
>> for us to hit one broker harder than the other, so ideally we'd like
>> to setup these nodes in a cluster. That way, we can have consumers
>> evenly distributed across brokers.
>>
>> We could create one queue that all consumers consume from, but I'm not
>> sure that would solve our problems since that queue (and the broker it
>> was created on) would become a hotspot. Right?
>>
>> So from that article, I like the idea of each node creating a queue,
>> e.g. myQueue.1, myQueue.2, etc. and having the producer round robin
>> them. What I don't know, is how the producer knows how many queues/
>> nodes there are. The producers could send to an exchange and just set
>> the routingKey, but still, I think the problem is the same.
>>
>> I guess I could use the management plugin, but I'm trying to figure
>> out how to do that without issuing a request for every message I
>> produce.
>>
>> I could imagine some kind of handshake when a producer comes online,
>> he sends a "who's-alive" message to a topic exchange and waits for
>> responses from consumers. Also, when a consumer comes online, he
>> immediately sends out a "I'm-alive" message. In a direct exchange
>> (either the default or one that I create), is a routingKey assumed to
>> be a queue name? If I made the queues auto_delete, what would happen
>> when the producer tries to send a message to a queue that no longer
>> exists? Would I get an exception such that I can catch it and send the
>> message to the next guy in the list?
>>
>> Thanks,
>> Justin
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>