<div dir="ltr">Steve,<div><br></div><div>Thanks for your continued help in the matter, you've done a great job at answering my questions. �I'm going to take your advice and run both performance and stress tests, starting with a prefetch count of 1, and then increasing to 2, and so on. �I think that the tests will help to highlight the integrity of my pipeline (i.e. bottlenecks, network latency, etc).</div>
<div><br></div><div style>Thanks again!</div><div style><br></div><div style>Kevin</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Feb 18, 2013 at 11:25 AM, Steve Powell <span dir="ltr"><<a href="mailto:steve@rabbitmq.com" target="_blank">steve@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Kevin,</div><div>I'm glad you managed to 'solve' your problem.</div>
<div><br></div><div>The acknowledgement (ACK) is to tell the RabbitMQ Server that the client is</div><div>willing to take responsibility for the message: RabbitMQ will not deliver this</div><div>message to anyone again. �If the channel closes/stops/dissappears and a</div>
<div>message sent to it hasn't been acknowledged, or if a message is rejected (with</div><div>requeue) �(NACKed with requeue), then RabbitMQ will attempt to deliver the</div><div>message again. �That may or may not be to the same consumer, channel or</div>
<div>client. �The prefetch count is a way of limiting the number of messages</div><div>RabbitMQ Server will allow unACKed (and unNACKed) on a channel.</div><div><br></div><div>Let's say you have one consumer on one channel on each client, and you have</div>
<div>two clients, A and B. Let's also say that A and B have prefetch count set to 2.�</div><div>We'll assume that A and B are served in that order by the RabbitMQ server.</div><div><br></div><div>If 5 messages arrive on the queue (m1..5) then the messages will arrive on the</div>
<div>QueueingConsumer client queues on A and B as follows:</div><div><br></div><div>A: �[m1, m3]</div><div>B: �[m2, m4]</div><div>and message m5 will stay in the server.</div><div><br></div><div>I presume both A and B will see a message and remove it from the</div>
<div>QueueingConsumer queue:</div><div><br></div><div>A: �[m3] �(processing m1)</div><div>B: �[m4] �(processing m2)</div><div><br></div><div>When the first message is processed successfully (either m1 on A, or m2 on B)</div>
<div>and the client sends ACK back to the server, then message m5 can be sent to</div><div>the client who ACKed first, let's say it is B:</div><div><br></div><div>A: �[m3] �(processing m1)</div><div>B: �[m4, m5] � (not processing a message)</div>
<div><br></div><div>But note that B might have already started processing m4 by this time (it</div><div>more probably would have, since the message m4 is already in the client),</div><div>so it would soon (or already)�look like this:</div>
<div><br></div><div>A: �[m3] �(processing m1)</div><div>B: �[m5] �(processing m4)</div><div><br></div><div>As messages arrive on the queue in the server, they are sent to either A or B</div><div>as they successfully (or otherwise) finish processing the previous messages.</div>
<div><br></div><div>Let's say that B finishes first again (ACKnowledging m4), but that �no more</div><div>messages arrive on the queue before then. �We then get into this situation:</div><div><br></div><div>A: �[m3] �(processing m1)</div>
<div>B: �[] �(processing m5)</div><div><br></div><div>B has an empty queue.</div><div><br></div><div>If B again finishes first (a fast and efficient machine, or a lightweight</div><div>message m5?) we could see this situation:</div>
<div><br></div><div>A: �[m3] �(processing m1)</div><div>B: �[] �(idle, waiting on internal queue)</div><div><br></div><div>A might take a very long time to process message m1, and m3 has to wait until</div><div>A has finished (or aborts) before it will be processed, even though B is</div>
<div>unoccupied and available.</div><div><br></div><div>The effects of the prefetch count >1 are therefore three-fold:</div><div><br></div><div>1) The internal client queues never hold more than prefetch-count messages,</div>
<div>� �all excess messages 'back-up' in the server.</div><div><br></div><div>2) Some messages can get 'caught' behind a message that takes a long time to</div><div>� �process: waiting indefinitely, even though there may be servers ready and</div>
<div>� �waiting�to process them.</div><div><br></div><div>3) Messages can be sent to a client while it is processing the previous</div><div>� �message, so the network transfer time can overlap processing time, reducing</div>
<div>� �the time spent idle in the client when there are messages to process.</div><div><br></div><div>Notice that there is a (potential) price to pay for the benefit in item 3. �If</div><div>you reduce the time spent waiting for network transfer time while idle, you</div>
<div>run the risk of processing some messages very late indeed.</div><div><br></div><div>As soon as you have multiple clients listening to the same queue you cannot</div><div>guarantee the order of processing of messages, whatever you set the prefetch</div>
<div>counts to.</div><div><br></div><div>You should be able to see what happens (can happen) with more clients, or with</div><div>a larger (or unlimited) prefetch count.</div><div><br></div><div>Whether you wish to use a setting like this very much depends upon how large</div>
<div>the network latency is, what the risk of message failures are, and what the</div><div>rate of message arrival might be (it will doubtless vary in any case).</div><div><br></div><div><div>A prefetch count of 2 and a variable number of clients works reasonably well:</div>
<div><br></div><div>- when order of message processing is not critical;</div><div>- when message processing is relatively reliable, and not too variable; and</div><div>- when the network transfer time is not too large per message (e.g. messages are</div>
<div>� reasonably small).</div><div><br></div><div>Outside of these parameters I recommend you run some careful tests.</div><div><br></div><div>In fact, let me recommend you run some careful performance and stress tests in</div>
<div>any case. �It is notoriously difficult to predict where the bottlenecks will</div><div>be in any particular configuration. �These bottlenecks will move depending</div><div>upon load, and sometimes when the load changes (up or down).</div>
<div><br></div><div>It may be, with low to medium message rates, that all this fuss makes no</div><div>difference over a prefetch count of 1. �You should check this before investing</div><div>in complex (and potentially buggy) code. You should also ensure that you</div>
</div><div>really need many processing clients and that you fully understand the</div><div>interactions between the processing that these clients undertake -- it</div><div>is possible that there are mutual blocks or other serialisation effects</div>
<div>that will have an impact upon the service rates (or even success)�</div><div>of your clients.</div><div><br></div><div>I hope I have begun to answer your question. �Good luck.</div><div>
<span style="border-collapse:separate;border-spacing:0px"><div style="word-wrap:break-word"><span style="font-style:normal;border-collapse:separate;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="font-family:Georgia;font-weight:normal;font-style:normal;font-size:medium;font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px;line-height:normal;border-collapse:separate"><div style="word-wrap:break-word">
<span style="font-size:12px;font-style:normal;font-family:Georgia;font-weight:normal">Steve Powell</span><span style="font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px;line-height:normal;border-collapse:separate"></span></div>
</span></div></span></div></span><div style="font-style:normal"><font face="Georgia" size="3"><span style="font-size:11px">[</span></font><i><font face="Georgia" size="3"><span style="font-size:11px">M</span></font></i><font face="Georgia" size="3"><span style="font-size:11px">: <a href="tel:%2B44-7815-838-558" value="+447815838558" target="_blank">+44-7815-838-558</a>;</span></font><font face="Georgia" size="3"><span style="font-size:11px">�</span></font><i><font face="Georgia" size="3"><span style="font-size:11px">H</span></font></i><font face="Georgia" size="3"><span style="font-size:11px">:<a href="tel:%2B44-1962-775-598" value="+441962775598" target="_blank">+44-1962-775-598</a>]</span></font></div>
<div style="font-style:normal;font-family:Helvetica;font-size:12px"><i><font face="Georgia"><span style="font-size:10px">Links:�</span></font><a href="http://www.springsource.org/" target="_blank"><font face="Georgia"><span style="font-size:10px">SpringSource</span></font></a></i><i><font face="Georgia"><span style="font-size:10px">�</span></font></i><i><font face="Georgia"><span style="font-size:10px">(</span></font></i><i><font face="Georgia"><span style="font-size:10px">a�division of�</span></font></i><a href="http://www.vmware.com/" target="_blank"><i><font face="Georgia"><span style="font-size:10px">VMware</span></font></i></a><i><font face="Georgia"><span style="font-size:10px">),</span></font></i><p style="display:inline!important">
<i><font face="Georgia"><span style="font-size:10px">�</span></font></i><a href="http://www.eclipse.org/virgo" target="_blank"><i><font face="Georgia"><span style="font-size:10px">Virgo</span></font></i></a><i><font face="Georgia"><span style="font-size:10px">,�</span></font></i></p>
<i><a href="http://www.rabbitmq.com" target="_blank"><font face="Georgia"><span style="font-size:10px">RabbitMQ</span></font></a><font face="Georgia"><span style="font-size:10px">.</span></font></i></div><div class="im">
<div style="font-style:normal;font-family:Helvetica;font-size:12px">
<i><font face="Georgia"><span style="font-size:10px">-----------------------------------------------------------------------</span></font></i></div><span style="border-collapse:separate;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="font-style:normal;font-family:Georgia;font-weight:normal;font-size:medium;font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px;line-height:normal;border-collapse:separate"><div style="word-wrap:break-word">
<span style="font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px;line-height:normal;border-collapse:separate"><div style="word-wrap:break-word">
<div style="font-family:Georgia;font-size:medium;font-weight:normal;font-style:normal"><span style="border-collapse:collapse;line-height:14px;font-size:10px"><font color="#3b3b3b">Good design:</font></span></div><div style="font-family:Georgia;font-size:medium;font-weight:normal;font-style:normal">
<span style="border-collapse:collapse;line-height:14px;font-size:10px"><font color="#3b3b3b">� �is innovative, useful, aesthetic;</font></span></div><div><span style="border-collapse:collapse;line-height:14px"><span style="font-size:10px"><font color="#3b3b3b">� �is understandable, unobtrusive, honest;</font></span></span></div>
</div></span></div></span><span style="font-style:normal;font-family:Georgia;font-weight:normal;font-size:medium;border-collapse:collapse;line-height:14px"><span style="font-size:10px"><font color="#3b3b3b">� �is long-lasting, thorough, environmentally friendly;</font></span></span><span style="font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">
<span style="font-variant:normal;letter-spacing:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word"><div style="border-collapse:separate;line-height:normal">
<span style="border-collapse:collapse;line-height:14px"><span style="font-family:Georgia;font-weight:normal;font-size:10px"><font color="#3b3b3b">� �and�</font></span></span><i style="color:rgb(59,59,59);font-size:10px;line-height:14px">is�as little design as possible.</i></div>
<div style="border-collapse:separate;line-height:normal"><span style="border-collapse:collapse;line-height:14px"><font style="font-style:normal;color:rgb(102,102,102);font-size:6px"><i>Copyright Dieter Rams, amended March 2003; October 2009; and August 2012</i></font></span></div>
</div></span></div></span></div></span></div></span></div></div></span>
</div><div class="im">
<br><div><div>On 12 Feb 2013, at 19:28, Kevin Behr <<a href="mailto:behrk2@gmail.com" target="_blank">behrk2@gmail.com</a>> wrote:</div><br><blockquote type="cite"><div style="font-family:Georgia;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<font face="arial, sans-serif">Regarding your statement above, I had not taken that into consideration. �I've coded my clients to �only acknowledge upon successful processing of the delivered message. �If one of our many web services or datastore connections goes down, then unacknowledged messages will start piling up quickly, and the messaging and queuing pipeline will essentially come to a halt. I would expect this to be the correct behavior, however, as I don't want to throw out messages until someone discovers that a service is down.</font></div>
<div style="font-family:Georgia;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<font face="arial, sans-serif"><br></font></div><div style="font-family:Georgia;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<font face="arial, sans-serif">As I will be using QueuingConsumer, I'm now wondering�<i>how�</i>I should determine�<i>what�</i>to set my prefetch counts to. �Do you have any further advice on this? �I'm still a little confused there.</font></div>
</blockquote></div><br></div></div></blockquote></div><br></div>