Then your original idea should work.<br><br>With scaling like this you should be concerned about one of the<br>consumers failing.� Something like having 2 or more consumers<br>for each of the 100 types of messages but maybe that's what<br>
you meant.<br clear="all"><br>Thanks,<br><br>- Jim<br><br>Jim Irrer� �� <a href="mailto:irrer@umich.edu">irrer@umich.edu</a>� � �� (734) 647-4409<br>University of Michigan Hospital Radiation Oncology<br>519 W. William St.� � � � � �� Ann Arbor, MI 48103<br>
<br><br><div class="gmail_quote">On Thu, Mar 24, 2011 at 9:30 AM, Mark Steele <span dir="ltr"><<a href="mailto:msteele@beringmedia.com">msteele@beringmedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
The idea here is that in a perfect world, I would have only 1 queue per exchange.�However that won't scale since queues only exists on the node on which they are declared.<div><br></div><div>To get around this, I want to create more queues across all the nodes, and distribute the load based on routing keys. The random pick of a routing key by the producer is simply to distribute load on the pre-created queues (that are spread across the nodes in the cluster).</div>
<div><br></div><div><div class="im"><br clear="all">Mark Steele<br>Bering Media Inc.<br><div><br></div>
<br><br></div><div><div></div><div class="h5"><div class="gmail_quote">On Wed, Mar 23, 2011 at 6:10 PM, Jim Irrer <span dir="ltr"><<a href="mailto:irrer@umich.edu" target="_blank">irrer@umich.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
To better understand - The 10 billion messages each need to be processed with the same<br>sort of processing?� The reason for having 10 direct exchanges would be that there are 10 different kinds of<br>messages, is that true, or was your intent to have the message producers have a<br>
role in choosing a consumer?<br clear="all"><br>Thanks,<br><br>- Jim<br><br>Jim Irrer� �� <a href="mailto:irrer@umich.edu" target="_blank">irrer@umich.edu</a>� � �� <a href="tel:%28734%29%20647-4409" target="_blank">(734) 647-4409</a><br>
University of Michigan Hospital Radiation Oncology<br>
519 W. William St.� � � � � �� Ann Arbor, MI 48103<br>
<br><br><div class="gmail_quote">On Wed, Mar 23, 2011 at 4:29 PM, Mark Steele <span dir="ltr"><<a href="mailto:msteele@beringmedia.com" target="_blank">msteele@beringmedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi folks,<div><br></div><div>I've been noodling a scale problem for a while, and wanted some input.</div><div><br></div><div>Here's my scenario:</div><div><br></div><div>I have a relatively small number of publishers (1000), who I expect will want to publish a very large number of messages (let's say 10 billion messages a day) to a about 10 exchanges.</div>
<div><br></div><div>My naive first stab at trying to get this to scale using rabbit would be�</div><div><ul><li>Create 10 direct exchanges</li><li>Create 100 queues to each of the exchanges (randomly distributed onto several nodes in a rabbit cluster).�</li>
<li>Bind each of the queues to a topic (random number from 1-100)</li><li>Attach consumers to each of these queues (insert load balancer here)</li><li>Have publishers randomly pick a number from 1-100 and use that as the routing key</li>
</ul></div><div>Does this approach make sense for getting scale?�</div><div><br></div><div>Cheers</div><div><br clear="all">Mark Steele<br>Bering Media Inc.<br><br>
</div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br>
</blockquote></div><br></div></div></div>
</blockquote></div><br>