[rabbitmq-discuss] Scaling Rabbit

[Mark Steele]

The idea here is that in a perfect world, I would have only 1 queue per
exchange. However that won't scale since queues only exists on the node on
which they are declared.

To get around this, I want to create more queues across all the nodes, and
distribute the load based on routing keys. The random pick of a routing key
by the producer is simply to distribute load on the pre-created queues (that
are spread across the nodes in the cluster).


Mark Steele
Bering Media Inc.



On Wed, Mar 23, 2011 at 6:10 PM, Jim Irrer <irrer at umich.edu> wrote:

> To better understand - The 10 billion messages each need to be processed
> with the same
> sort of processing?  The reason for having 10 direct exchanges would be
> that there are 10 different kinds of
> messages, is that true, or was your intent to have the message producers
> have a
> role in choosing a consumer?
>
> Thanks,
>
> - Jim
>
> Jim Irrer     irrer at umich.edu       (734) 647-4409
> University of Michigan Hospital Radiation Oncology
> 519 W. William St.             Ann Arbor, MI 48103
>
>
> On Wed, Mar 23, 2011 at 4:29 PM, Mark Steele <msteele at beringmedia.com>wrote:
>
>> Hi folks,
>>
>> I've been noodling a scale problem for a while, and wanted some input.
>>
>> Here's my scenario:
>>
>> I have a relatively small number of publishers (1000), who I expect will
>> want to publish a very large number of messages (let's say 10 billion
>> messages a day) to a about 10 exchanges.
>>
>> My naive first stab at trying to get this to scale using rabbit would be
>>
>>    - Create 10 direct exchanges
>>    - Create 100 queues to each of the exchanges (randomly distributed
>>    onto several nodes in a rabbit cluster).
>>    - Bind each of the queues to a topic (random number from 1-100)
>>    - Attach consumers to each of these queues (insert load balancer here)
>>    - Have publishers randomly pick a number from 1-100 and use that as
>>    the routing key
>>
>> Does this approach make sense for getting scale?
>>
>> Cheers
>>
>> Mark Steele
>> Bering Media Inc.
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110324/0eb7300a/attachment.htm>