<div dir="ltr">Steve,<div><br></div><div>Thanks for your continued help in the matter, you've done a great job at answering my questions. I'm going to take your advice and run both performance and stress tests, starting with a prefetch count of 1, and then increasing to 2, and so on. I think that the tests will help to highlight the integrity of my pipeline (i.e. bottlenecks, network latency, etc).</div>
<div><br></div><div style>Thanks again!</div><div style><br></div><div style>Kevin</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Feb 18, 2013 at 11:25 AM, Steve Powell <span dir="ltr"><<a href="mailto:steve@rabbitmq.com" target="_blank">steve@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Kevin,</div><div>I'm glad you managed to 'solve' your problem.</div>
<div><br></div><div>The acknowledgement (ACK) is to tell the RabbitMQ Server that the client is</div><div>willing to take responsibility for the message: RabbitMQ will not deliver this</div><div>message to anyone again. If the channel closes/stops/dissappears and a</div>
<div>message sent to it hasn't been acknowledged, or if a message is rejected (with</div><div>requeue) (NACKed with requeue), then RabbitMQ will attempt to deliver the</div><div>message again. That may or may not be to the same consumer, channel or</div>
<div>client. The prefetch count is a way of limiting the number of messages</div><div>RabbitMQ Server will allow unACKed (and unNACKed) on a channel.</div><div><br></div><div>Let's say you have one consumer on one channel on each client, and you have</div>
<div>two clients, A and B. Let's also say that A and B have prefetch count set to 2. </div><div>We'll assume that A and B are served in that order by the RabbitMQ server.</div><div><br></div><div>If 5 messages arrive on the queue (m1..5) then the messages will arrive on the</div>
<div>QueueingConsumer client queues on A and B as follows:</div><div><br></div><div>A: [m1, m3]</div><div>B: [m2, m4]</div><div>and message m5 will stay in the server.</div><div><br></div><div>I presume both A and B will see a message and remove it from the</div>
<div>QueueingConsumer queue:</div><div><br></div><div>A: [m3] (processing m1)</div><div>B: [m4] (processing m2)</div><div><br></div><div>When the first message is processed successfully (either m1 on A, or m2 on B)</div>
<div>and the client sends ACK back to the server, then message m5 can be sent to</div><div>the client who ACKed first, let's say it is B:</div><div><br></div><div>A: [m3] (processing m1)</div><div>B: [m4, m5] (not processing a message)</div>
<div><br></div><div>But note that B might have already started processing m4 by this time (it</div><div>more probably would have, since the message m4 is already in the client),</div><div>so it would soon (or already) look like this:</div>
<div><br></div><div>A: [m3] (processing m1)</div><div>B: [m5] (processing m4)</div><div><br></div><div>As messages arrive on the queue in the server, they are sent to either A or B</div><div>as they successfully (or otherwise) finish processing the previous messages.</div>
<div><br></div><div>Let's say that B finishes first again (ACKnowledging m4), but that no more</div><div>messages arrive on the queue before then. We then get into this situation:</div><div><br></div><div>A: [m3] (processing m1)</div>
<div>B: [] (processing m5)</div><div><br></div><div>B has an empty queue.</div><div><br></div><div>If B again finishes first (a fast and efficient machine, or a lightweight</div><div>message m5?) we could see this situation:</div>
<div><br></div><div>A: [m3] (processing m1)</div><div>B: [] (idle, waiting on internal queue)</div><div><br></div><div>A might take a very long time to process message m1, and m3 has to wait until</div><div>A has finished (or aborts) before it will be processed, even though B is</div>
<div>unoccupied and available.</div><div><br></div><div>The effects of the prefetch count >1 are therefore three-fold:</div><div><br></div><div>1) The internal client queues never hold more than prefetch-count messages,</div>
<div> all excess messages 'back-up' in the server.</div><div><br></div><div>2) Some messages can get 'caught' behind a message that takes a long time to</div><div> process: waiting indefinitely, even though there may be servers ready and</div>
<div> waiting to process them.</div><div><br></div><div>3) Messages can be sent to a client while it is processing the previous</div><div> message, so the network transfer time can overlap processing time, reducing</div>
<div> the time spent idle in the client when there are messages to process.</div><div><br></div><div>Notice that there is a (potential) price to pay for the benefit in item 3. If</div><div>you reduce the time spent waiting for network transfer time while idle, you</div>
<div>run the risk of processing some messages very late indeed.</div><div><br></div><div>As soon as you have multiple clients listening to the same queue you cannot</div><div>guarantee the order of processing of messages, whatever you set the prefetch</div>
<div>counts to.</div><div><br></div><div>You should be able to see what happens (can happen) with more clients, or with</div><div>a larger (or unlimited) prefetch count.</div><div><br></div><div>Whether you wish to use a setting like this very much depends upon how large</div>
<div>the network latency is, what the risk of message failures are, and what the</div><div>rate of message arrival might be (it will doubtless vary in any case).</div><div><br></div><div><div>A prefetch count of 2 and a variable number of clients works reasonably well:</div>
<div><br></div><div>- when order of message processing is not critical;</div><div>- when message processing is relatively reliable, and not too variable; and</div><div>- when the network transfer time is not too large per message (e.g. messages are</div>
<div> reasonably small).</div><div><br></div><div>Outside of these parameters I recommend you run some careful tests.</div><div><br></div><div>In fact, let me recommend you run some careful performance and stress tests in</div>
<div>any case. It is notoriously difficult to predict where the bottlenecks will</div><div>be in any particular configuration. These bottlenecks will move depending</div><div>upon load, and sometimes when the load changes (up or down).</div>
<div><br></div><div>It may be, with low to medium message rates, that all this fuss makes no</div><div>difference over a prefetch count of 1. You should check this before investing</div><div>in complex (and potentially buggy) code. You should also ensure that you</div>
</div><div>really need many processing clients and that you fully understand the</div><div>interactions between the processing that these clients undertake -- it</div><div>is possible that there are mutual blocks or other serialisation effects</div>
<div>that will have an impact upon the service rates (or even success) </div><div>of your clients.</div><div><br></div><div>I hope I have begun to answer your question. Good luck.</div><div>
<span style="border-collapse:separate;border-spacing:0px"><div style="word-wrap:break-word"><span style="font-style:normal;border-collapse:separate;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="font-family:Georgia;font-weight:normal;font-style:normal;font-size:medium;font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px;line-height:normal;border-collapse:separate"><div style="word-wrap:break-word">
<span style="font-size:12px;font-style:normal;font-family:Georgia;font-weight:normal">Steve Powell</span><span style="font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px;line-height:normal;border-collapse:separate"></span></div>
</span></div></span></div></span><div style="font-style:normal"><font face="Georgia" size="3"><span style="font-size:11px">[</span></font><i><font face="Georgia" size="3"><span style="font-size:11px">M</span></font></i><font face="Georgia" size="3"><span style="font-size:11px">: <a href="tel:%2B44-7815-838-558" value="+447815838558" target="_blank">+44-7815-838-558</a>;</span></font><font face="Georgia" size="3"><span style="font-size:11px"> </span></font><i><font face="Georgia" size="3"><span style="font-size:11px">H</span></font></i><font face="Georgia" size="3"><span style="font-size:11px">:<a href="tel:%2B44-1962-775-598" value="+441962775598" target="_blank">+44-1962-775-598</a>]</span></font></div>
<div style="font-style:normal;font-family:Helvetica;font-size:12px"><i><font face="Georgia"><span style="font-size:10px">Links: </span></font><a href="http://www.springsource.org/" target="_blank"><font face="Georgia"><span style="font-size:10px">SpringSource</span></font></a></i><i><font face="Georgia"><span style="font-size:10px"> </span></font></i><i><font face="Georgia"><span style="font-size:10px">(</span></font></i><i><font face="Georgia"><span style="font-size:10px">a division of </span></font></i><a href="http://www.vmware.com/" target="_blank"><i><font face="Georgia"><span style="font-size:10px">VMware</span></font></i></a><i><font face="Georgia"><span style="font-size:10px">),</span></font></i><p style="display:inline!important">
<i><font face="Georgia"><span style="font-size:10px"> </span></font></i><a href="http://www.eclipse.org/virgo" target="_blank"><i><font face="Georgia"><span style="font-size:10px">Virgo</span></font></i></a><i><font face="Georgia"><span style="font-size:10px">, </span></font></i></p>
<i><a href="http://www.rabbitmq.com" target="_blank"><font face="Georgia"><span style="font-size:10px">RabbitMQ</span></font></a><font face="Georgia"><span style="font-size:10px">.</span></font></i></div><div class="im">
<div style="font-style:normal;font-family:Helvetica;font-size:12px">
<i><font face="Georgia"><span style="font-size:10px">-----------------------------------------------------------------------</span></font></i></div><span style="border-collapse:separate;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="font-style:normal;font-family:Georgia;font-weight:normal;font-size:medium;font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px;line-height:normal;border-collapse:separate"><div style="word-wrap:break-word">
<span style="font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px;line-height:normal;border-collapse:separate"><div style="word-wrap:break-word">
<div style="font-family:Georgia;font-size:medium;font-weight:normal;font-style:normal"><span style="border-collapse:collapse;line-height:14px;font-size:10px"><font color="#3b3b3b">Good design:</font></span></div><div style="font-family:Georgia;font-size:medium;font-weight:normal;font-style:normal">
<span style="border-collapse:collapse;line-height:14px;font-size:10px"><font color="#3b3b3b"> is innovative, useful, aesthetic;</font></span></div><div><span style="border-collapse:collapse;line-height:14px"><span style="font-size:10px"><font color="#3b3b3b"> is understandable, unobtrusive, honest;</font></span></span></div>
</div></span></div></span><span style="font-style:normal;font-family:Georgia;font-weight:normal;font-size:medium;border-collapse:collapse;line-height:14px"><span style="font-size:10px"><font color="#3b3b3b"> is long-lasting, thorough, environmentally friendly;</font></span></span><span style="font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word"><div style="border-collapse:separate;line-height:normal">
<span style="border-collapse:collapse;line-height:14px"><span style="font-family:Georgia;font-weight:normal;font-size:10px"><font color="#3b3b3b"> and </font></span></span><i style="color:rgb(59,59,59);font-size:10px;line-height:14px">is as little design as possible.</i></div>
<div style="border-collapse:separate;line-height:normal"><span style="border-collapse:collapse;line-height:14px"><font style="font-style:normal;color:rgb(102,102,102);font-size:6px"><i>Copyright Dieter Rams, amended March 2003; October 2009; and August 2012</i></font></span></div>
</div></span></div></span></div></span></div></span></div></div></span>
</div><div class="im">
<br><div><div>On 12 Feb 2013, at 19:28, Kevin Behr <<a href="mailto:behrk2@gmail.com" target="_blank">behrk2@gmail.com</a>> wrote:</div><br><blockquote type="cite"><div style="font-family:Georgia;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<font face="arial, sans-serif">Regarding your statement above, I had not taken that into consideration. I've coded my clients to only acknowledge upon successful processing of the delivered message. If one of our many web services or datastore connections goes down, then unacknowledged messages will start piling up quickly, and the messaging and queuing pipeline will essentially come to a halt. I would expect this to be the correct behavior, however, as I don't want to throw out messages until someone discovers that a service is down.</font></div>
<div style="font-family:Georgia;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<font face="arial, sans-serif"><br></font></div><div style="font-family:Georgia;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<font face="arial, sans-serif">As I will be using QueuingConsumer, I'm now wondering <i>how </i>I should determine <i>what </i>to set my prefetch counts to. Do you have any further advice on this? I'm still a little confused there.</font></div>
</blockquote></div><br></div></div></blockquote></div><br></div>