<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Oops, I meant to send this to the ML. Forwarding...<div><br></div><div>cr<br><div><br><div>Begin forwarded message:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" color="#000000" style="font: 12.0px Helvetica; color: #000000"><b>From: </b></font><font face="Helvetica" size="3" style="font: 12.0px Helvetica">Chuck Remes <<a href="mailto:cremes.devlist@mac.com">cremes.devlist@mac.com</a>></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" color="#000000" style="font: 12.0px Helvetica; color: #000000"><b>Date: </b></font><font face="Helvetica" size="3" style="font: 12.0px Helvetica">June 4, 2009 4:16:17 PM CDT</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" color="#000000" style="font: 12.0px Helvetica; color: #000000"><b>To: </b></font><font face="Helvetica" size="3" style="font: 12.0px Helvetica">smittycb10 <<a href="mailto:msmith1638@gmail.com">msmith1638@gmail.com</a>></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" color="#000000" style="font: 12.0px Helvetica; color: #000000"><b>Subject: </b></font><font face="Helvetica" size="3" style="font: 12.0px Helvetica"><b>Re: [rabbitmq-discuss] rabbit MQ high-through put stock quotes</b></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div> </div><div><br>On Jun 4, 2009, at 2:48 PM, smittycb10 wrote:<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">This is a follow up to my previous message with some more details about how I<br></blockquote><blockquote type="cite">envision using Rabbit MQ.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Currently I have a system where a server takes in all messages from our data<br></blockquote><blockquote type="cite">provider and then streams the messages via UDP to a consumer application.<br></blockquote><blockquote type="cite">This set up can handle the message load but is proving to be difficult and<br></blockquote><blockquote type="cite">costly to scale. One producer to one Consuming server which then filters<br></blockquote><blockquote type="cite">data to desktop clients.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">My idea for a future system is to have the same server simply push the<br></blockquote><blockquote type="cite">messages to a rabbit MQ Broker or Brokers and then have all clients simply<br></blockquote><blockquote type="cite">listen to whatever queues they need to get there jobs done. I need to design<br></blockquote><blockquote type="cite">for 100,000 (one hundred thousand) messages to made available for consuming<br></blockquote><blockquote type="cite">per second. Most consumers will only consume a fraction of these messages,<br></blockquote><blockquote type="cite">although some of our server pieces may need to subscribe to the whole<br></blockquote><blockquote type="cite">universe of incoming messages. Messages can and will be batched on the<br></blockquote><blockquote type="cite">publishing side.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">So the use case would be, one publisher pushing messages to the "Broker<br></blockquote><blockquote type="cite">Cloud" many consumers some needing to listen to all messages others would<br></blockquote><blockquote type="cite">only be interested in a particular Stock symbol at a time.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I will experiment with some fanout experiments next, any advice or direction<br></blockquote><blockquote type="cite">you have would be greatly appreciated. In a fanout pattern if say a quote<br></blockquote><blockquote type="cite">comes in for IBM how do I distribute only to people interested in IBM?<br></blockquote><br>Let me take a crack at this. Be forewarned that everything I write here might be wrong. :)<br><br>We have 3 kinds of queues to choose from: direct, fanout and topic.<br><br>In terms of performance, they behave in the order listed above from fastest to slowest.<br><br>For direct exchanges, you bind your queue to it with a specific routing key. Any messages published to that exchange need to include a routing key. If the message's routing key *exactly* matches your binding's routing key, the message is delivered to your queue.<br><br>For fanout exchanges, you bind a queue to it without any routing key. Any messages published to it are *unconditionally* delivered to the queue. The reason I didn't mark this as the fastest queue type is because you generally only pick it when you have lots of bound queues so the message needs to be copied to each. (Of course, this is implementation specific and could be very fast or slow; I don't know how Rabbit performs this operation.) If you only have one subscriber, this will be faster than the direct exchange because you skip the routing key comparison.<br><br>For topic exchanges, you bind a queue to the exchange with a routing pattern. Any messages published with a key that matches your pattern are delivered to the queue.<br><br>With that out of the way, I would suggest a fanout exchange per high-volume symbol. Any interested subscribers can bind a uniquely-named queue to that exchange and get all messages for IBM. Note I said you need a uniquely-named queue per subscriber. If you use a single queue for all subscribers, the subscribers will get messages in a round-robin fashion (usually). So N subscribers would only see an update every N messages which is probably not what you want.<br><br>Alternately, you could use a direct exchange for the low-volume stocks. Interested subscribers could bind their (again) uniquely-named queue with a routing key matching the symbol (or some other identifier). Rabbit is smart enough that it will only deliver messages to queues where the keys match exactly. The reason I suggest this is due to potential performance concerns on the server side if it has to handle hundreds or thousands of unique exchanges. I don't know if Rabbit suffers from any performance degradation in this situation, so make sure to benchmark it.<br><br>Additionally, you'll want to publish messages with "nowait" and "noack" and "not persistent" and make sure the queues are not "durable." See the API docs for your chosen library to see how to set those flags. "Nowait" means the client should not wait for a response to the publish method. "No ack" means the message itself doesn't need to be acknowledged as received; the server will "pre-ack" each message pulled from the queue. And the "not persistent" setting interacts with the "durability" setting on the queue. Durable messages (inside durable queues) are stored on disk so that a crash and restart doesn't lose any messages. For transient messages like stock quotes, you don't want them persisted at this level.<br><br>Someone else on the list will have to answer your questions about scaling up to 100k/sec. I think Erlang processes are still single-threaded so you probably need to run multiple instances to take advantage of multi-core servers. Perhaps this is Rabbit clustering... I honestly don't know.<br><br>FYI, tests on my hardware show that publishing from one process to another on the same box using a direct exchange and the suggestions listed above introduces around 600 microseconds in delivery latency. Pay close attention to your serialization/deserialization code; its performance could be a gating factor.<br><br>Let us know how things turn out.<br><br>cr<br><br></div></blockquote></div><br></div></body></html>