[rabbitmq-discuss] questions about distributed queue

Tue Aug 18 09:23:37 BST 2009

Brian

On Mon, Aug 17, 2009 at 8:38 PM, Brian Mitchell<binary42 at gmail.com> wrote:
> On Mon, Aug 17, 2009 at 14:32, Alexis
> Richardson<alexis.richardson at gmail.com> wrote:
>> On Mon, Aug 17, 2009 at 5:22 PM, Paul Dix<paul at pauldix.net> wrote:
>>> So what exactly does option 1 look like?
>>>
>>> It sounds like it's possible to have a queue with the same id on two
>>> different nodes bound to the same exchange.
>>
>> Not quite.  Same routing - two queues, two ids.  Actually now that I
>> think about it that won't give you exactly what you need.  More below.
>
> Right. I figured the producers would round robin their production in
> some fashion for that method. Talking to Paul on IRC made it clear
> that it would be ideal to limit the amount of logic and load balancing
> that the producer and/or consumer is responsible for.

That sounds right to me, based on what Paul's been saying he needs.

> Another idea I've played with is splitting the vhosts or clusters up
> and optionally only replicating internally on each one to a reasonable
> limit for availability control. In this case, sharding methods like
> bucket selection could be used to assign producers to a single cluster
> or vhost combo taking the need to cycle out. Consumers could either
> connect to a single cluster or vhost or many as appropriate for the
> flow characteristics expected.

Yes, although I think you would need high loads to justify the
additional management overhead.

> The only plusses of this are the fact that you can scale complex
> exchange/queue setups w/o having to generate and track a lot of unique
> routing keys that need to be coordinated with the consumer side.
> Instead you just have a list of hosts that can be used.

Yes.

> I've also
> thought of a few ways one could put a load balancer in front of this
> quite easily on either side if there is a concern with how smart a
> client must be.

IMO this is a good way to go.

alexis