[rabbitmq-discuss] Scaling Rabbit

[Jim Irrer]

Then your original idea should work.

With scaling like this you should be concerned about one of the
consumers failing.  Something like having 2 or more consumers
for each of the 100 types of messages but maybe that's what
you meant.

Thanks,

- Jim

Jim Irrer     irrer at umich.edu       (734) 647-4409
University of Michigan Hospital Radiation Oncology
519 W. William St.             Ann Arbor, MI 48103


On Thu, Mar 24, 2011 at 9:30 AM, Mark Steele <msteele at beringmedia.com>wrote:

> The idea here is that in a perfect world, I would have only 1 queue per
> exchange. However that won't scale since queues only exists on the node on
> which they are declared.
>
> To get around this, I want to create more queues across all the nodes, and
> distribute the load based on routing keys. The random pick of a routing key
> by the producer is simply to distribute load on the pre-created queues (that
> are spread across the nodes in the cluster).
>
>
> Mark Steele
> Bering Media Inc.
>
>
>
> On Wed, Mar 23, 2011 at 6:10 PM, Jim Irrer <irrer at umich.edu> wrote:
>
>> To better understand - The 10 billion messages each need to be processed
>> with the same
>> sort of processing?  The reason for having 10 direct exchanges would be
>> that there are 10 different kinds of
>> messages, is that true, or was your intent to have the message producers
>> have a
>> role in choosing a consumer?
>>
>> Thanks,
>>
>> - Jim
>>
>> Jim Irrer     irrer at umich.edu       (734) 647-4409
>> University of Michigan Hospital Radiation Oncology
>> 519 W. William St.             Ann Arbor, MI 48103
>>
>>
>> On Wed, Mar 23, 2011 at 4:29 PM, Mark Steele <msteele at beringmedia.com>wrote:
>>
>>> Hi folks,
>>>
>>> I've been noodling a scale problem for a while, and wanted some input.
>>>
>>> Here's my scenario:
>>>
>>> I have a relatively small number of publishers (1000), who I expect will
>>> want to publish a very large number of messages (let's say 10 billion
>>> messages a day) to a about 10 exchanges.
>>>
>>> My naive first stab at trying to get this to scale using rabbit would be
>>>
>>>    - Create 10 direct exchanges
>>>    - Create 100 queues to each of the exchanges (randomly distributed
>>>    onto several nodes in a rabbit cluster).
>>>    - Bind each of the queues to a topic (random number from 1-100)
>>>    - Attach consumers to each of these queues (insert load balancer
>>>    here)
>>>    - Have publishers randomly pick a number from 1-100 and use that as
>>>    the routing key
>>>
>>> Does this approach make sense for getting scale?
>>>
>>> Cheers
>>>
>>> Mark Steele
>>> Bering Media Inc.
>>>
>>>
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110324/34553d67/attachment.htm>