<div dir="ltr">Many thanks Steve! A excellent summary. I've updated my blog post. Thanks especially for clarifying how the threading around consumers works. I'm using the .NET client. Is it a straight port of the Java client, or are there significant differences?<div>
<br></div><div>Cheers</div><div>Mike</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 24, 2013 at 11:30 AM, Tim Watson <span dir="ltr"><<a href="mailto:tim@rabbitmq.com" target="_blank">tim@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">A superlative follow up - thanks for clearing up the details Steve, and for the history lesson (which I for one, found very enlightening)!<div>
<br></div><div>Cheers,</div><div>Tim <div><div class="h5"><div><br><div><div>On 24 Sep 2013, at 10:57, Steve Powell wrote:</div><br><blockquote type="cite"><div style="word-wrap:break-word"><div>(Previous message sent before finalisation -- please ignore it.)</div>
<div><br></div><div>Sorry to butt in, but I wanted to clarify a few things (about the Java client):</div><div><br></div><div><span style="text-align:left">There is </span><i style="text-align:left">no</i><span style="text-align:left"> ‘rule’ “you shouldn’t share a channel between threads”. It is (good) advice, though.</span></div>
<div><span style="text-align:left"><br></span></div><div><span style="text-align:left">It is </span><i style="text-align:left">not</i><span style="text-align:left"> true that “</span><font face="American Typewriter" style="text-align:left">basic.deliver</font><span style="text-align:left"> will be called on the client library’s internal message loop thread”, <font face="American Typewriter">handleDelivery</font> is called on a <font face="American Typewriter">Consumer</font> callback thread.</span></div>
<div><span style="text-align:left"><br></span></div><div><span style="text-align:left">It is fine to call <font face="American Typewriter">ack</font> from a <font face="American Typewriter">Consumer</font>.<font face="American Typewriter">handleDelivery</font> method, and it is OK to call any other channel method, <i>but</i> it is hard to program multi-thread control of a channel without getting into serious difficulties.</span></div>
<div><span style="text-align:left"><br></span></div><div><span style="text-align:left">Some channel commands don't wait for a response, and some do. If you are waiting for a response then your application thread is waiting (blocked). If you issue a second command that wants to wait for a response, the Java Client will block the second thread until the response has returned from the first. This is thread-safe, and as designed. However, there is no serialisation of these requests: if many simultaneously issue commands, they may execute in any order. The state of the channel each command sees is therefore not predictable, and so the application threads need to cope with potential failures and reorderings.</span></div>
<div><span style="text-align:left"><br></span></div><div><span style="text-align:left">Even commands that don't expect a response can interfere with another thread's commands by changing the state of the channel, so it is quite important that the application threads know what each other are doing.</span></div>
<div><span style="text-align:left"><br></span></div><div><span style="text-align:left">Issuing <font face="American Typewriter">ack</font>s from many threads runs the risk of a double <font face="American Typewriter">ack</font> (<font face="American Typewriter">ack</font>ing the same message twice), which is a protocol error. </span><span style="text-align:left">Multi-<font face="American Typewriter">ack</font>s suffer from the same problem.</span></div>
<div><span style="text-align:left"><br></span></div>I refer you to the following sections on the Java Client API documentation page (<a href="http://www.rabbitmq.com/api-guide.html" target="_blank">http://www.rabbitmq.com/api-guide.html</a><span style="font-size:14px">):</span><div>
<h2 style="line-height:18px;text-align:left;color:rgb(255,102,0);font-size:24px;margin:20px 0px 0.5em;font-family:Helvetica,Arial,sans-serif;padding:0px"></h2><blockquote type="cite"><h2 style="line-height:18px;text-align:left;color:rgb(255,102,0);font-size:1.7em;margin:20px 0px 0.5em;font-family:Helvetica,Arial,sans-serif;padding:0px">
Channel thread-safety</h2></blockquote></div><blockquote type="cite"><p style="clear:left;line-height:18px;text-align:left;color:rgb(85,85,85);font-size:13px;font-family:Verdana,sans-serif"><span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Channel</span> instances are safe for use by multiple threads. Requests into a<span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Channel</span> are serialized, with only one thread being able to run a command on the <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Channel</span> at a time. Even so, applications should prefer using a <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Channel</span> per thread instead of sharing the same <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Channel</span> across multiple threads.</p>
</blockquote><div>and, in the following section:</div><div></div><blockquote type="cite"><p style="clear:left;line-height:18px;text-align:left;color:rgb(85,85,85);font-size:13px;font-family:Verdana,sans-serif">Callbacks to <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Consumer</span>s are dispatched on a thread separate from the thread managed by the <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Connection</span>. This means that <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Consumer</span>s can safely call blocking methods on the <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Connection</span> or <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Channel</span>, such as<span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">queueDeclare</span>, <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">txCommit</span>, <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">basicCancel</span> or <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">basicPublish</span>.</p>
<p style="clear:left;line-height:18px;text-align:left;color:rgb(85,85,85);font-size:13px;font-family:Verdana,sans-serif">Each <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Channel</span> has its own dispatch thread. For the most common use case of one <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Consumer</span> per <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Channel</span>, this means <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Consumer</span>s do not hold up other <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Consumer</span>s. If you have multiple <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Consumer</span>s per <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Channel</span> be aware that a long-running <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Consumer</span> may hold up dispatch of callbacks to other <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Consumer</span>s on that <span style="color:rgb(51,51,51);font-family:'Courier New',Courier,monospace;font-size:small;white-space:nowrap">Channel</span>.</p>
</blockquote><div style="text-align:left">If you are interested…. (background info):</div><div style="text-align:left"><br></div><div style="text-align:left">It <i>used to be the case</i> that the consumer callbacks (e.g. <font face="American Typewriter">handleDelivery</font>) executed on the <font face="American Typewriter">Connection</font> thread. (This is no longer true—see below.) This meant it was quite dangerous to issue channel commands in a callback method: the same thread that listened for frames from the server, might block waiting for a response from a channel command. <i>Deadlock.</i></div>
<div style="text-align:left"><br></div><div style="text-align:left">The <font face="American Typewriter">basic.ack</font> command was alright, because there is no response waited for (and any other command which doesn’t have a protocol response is fine, too). However, many applications tried to issue channel commands (like <font face="American Typewriter">declareQueue</font> or <font face="American Typewriter">createChannel</font>) which <i>did</i> expect responses, so they ran into trouble.</div>
<div style="text-align:left"><br></div><div style="text-align:left">To obviate this problem, the <font face="American Typewriter">QueueingConsumer</font> was created: this essentially does all the necessary things in the callback to receive the message, puts it in a (communicating, thread-safe) Java queue, and returns. The message can then be obtained from the Java queue by any other thread, including the original application thread. If channel commands are subsequently issued they cannot block the <font face="American Typewriter">Connection</font> thread.</div>
<div style="text-align:left"><br></div><div style="text-align:left">Issuing channel commands from more than one thread ran the risk of issuing a command requiring a response (a ‘synchronous’ command) while another thread was already waiting for a response. This would be an error (a thread-safety error) because the client cannot cope with overlapping synchronous commands on the same channel. This is prevented: a thread that issues a command expecting a response will block until any outstanding response is received on that channel.</div>
<div style="text-align:left"><br></div><div style="text-align:left">Although the thread-safety issues are (should be—let’s not get complacent) now all fixed, this solution just made a deadlock on the <font face="American Typewriter">Connection</font> thread more likely if you were trying to code your own <font face="American Typewriter">Consumer</font>.</div>
<div style="text-align:left"><div><br></div><div>In RabbitMQ 2.7.0 and 2.7.1, the Java client was changed to take the callback executions off of the <font face="American Typewriter">Connection</font> thread altogether. This was for three reasons. <i>One</i> was to allow channel commands to be issued in <font face="American Typewriter">handleDelivery</font> (<i>et al.</i>), which, by the way, meant that bespoke <font face="American Typewriter">Consumer</font>s were easier to write—<font face="American Typewriter">QueueingConsumer</font> is now not (so) necessary. The <i>second</i> was to recover more gracefully from <font face="American Typewriter">Consumer</font>s that do silly things: throwing exceptions, hanging indefinitely, and so on. And the <i>third</i> was to prevent <font face="American Typewriter">Consumer</font>s on <i>distinct</i> channels from holding each other up. (There is now a pool of threads used to execute <font face="American Typewriter">Consumer</font> callbacks, each channel’s callbacks being executed serially, but distinct channels’ callbacks being allowed to run concurrently.)</div>
<div><br></div></div><div style="text-align:left">It is quite safe to issue an <font face="American Typewriter">ack</font> (or <font face="American Typewriter">nack</font>) from another thread. But you should beware of doing this twice, so multi-threaded applications need to be carefully designed to avoid ‘confusion’ over which thread does what—“channel control fights”.</div>
<div style="text-align:left"><br></div><div style="text-align:left">Channel commands can be issued from multiple threads if you can ensure that the correct channel protocol is observed. To do this properly requires communication between the threads in the application, which can be non-trivial to get right (I recommend <i>Java Concurrency in Practice</i>, Addison Wesley, Goetz, et al.).</div>
<div style="text-align:left"><br></div><div style="text-align:left">So, the ‘<i>advice</i>’ is <i>not</i> to share channels between application threads, but it is <i>just advice</i>. It is expected that you might issue (<font face="American Typewriter">n</font>)<font face="American Typewriter">ack</font>s from the <font face="American Typewriter">Consumer</font> <font face="American Typewriter">handleDelivery</font> method, and put other <font face="American Typewriter">Channel</font> method calls there, too. No rules are being ‘broken’.</div>
<div style="text-align:left"><br></div><div>
<div style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:-webkit-auto;font-variant:normal;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:-webkit-auto;font-variant:normal;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<div style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:-webkit-auto;font-variant:normal;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:-webkit-auto;font-variant:normal;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<div style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:-webkit-auto;font-variant:normal;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><span style="border-collapse:separate;border-spacing:0px"><div style="word-wrap:break-word">
<span style="border-collapse:separate;border-spacing:0px"><div style="word-wrap:break-word"><span style="border-spacing:0px;border-collapse:separate"><div style="word-wrap:break-word"><span style="font-family:Georgia;font-size:12px;font-weight:normal;font-style:normal">Steve Powell </span><font face="Georgia" size="3" style="font-family:Georgia;font-size:medium;font-weight:normal;font-style:normal"><span style="font-size:11px">[</span></font><i style="font-family:Georgia;font-size:medium;font-weight:normal;font-style:normal"><font face="Georgia" size="3"><span style="font-size:11px">Cell</span></font></i><font face="Georgia" size="3" style="font-family:Georgia;font-size:medium;font-weight:normal;font-style:normal"><span style="font-size:11px">: <a href="tel:%2B44-7815-838-558" value="+447815838558" target="_blank">+44-7815-838-558</a></span></font><font face="Georgia" style="font-family:Georgia;font-weight:normal;font-style:normal"><span style="font-size:11px">]<span> </span></span><span style="font-size:10px">[</span></font><span style="font-size:10px"><a href="http://www.rabbitmq.com/" target="_blank">RabbitMQ</a>,<font face="Helvetica"><i> </i></font><a href="http://gopivotal.com/" target="_blank">Pivotal</a>]</span></div>
<div style="font-family:Georgia;font-size:medium;font-weight:normal;font-style:normal;word-wrap:break-word"><i style="font-size:12px;line-height:19.1875px">“L’enfer, c’est les autres.” </i><span style="line-height:19.1875px;font-size:12px">Sartre</span></div>
</span></div></span></div></span></div></div></div></div></div>
</div>
<br><div><div>On 23 Sep 2013, at 15:52, Mike Hadlow <<a href="mailto:mike@suteki.co.uk" target="_blank">mike@suteki.co.uk</a>> wrote:</div><br><blockquote type="cite"><div dir="ltr">Hi Michael, Tim,<div><br></div><div>
I've put your comments into a blog post: <a href="http://mikehadlow.blogspot.co.uk/2013/09/rabbitmq-amqp-channel-best-practices.html" target="_blank">http://mikehadlow.blogspot.co.uk/2013/09/rabbitmq-amqp-channel-best-practices.html</a>.</div>
<div><br></div><div>Let me know if there's anything you'd like to add/change/remove.</div><div><br></div><div>Many thanks</div><div>Mike</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Sep 23, 2013 at 12:35 PM, Michael Klishin <span dir="ltr"><<a href="mailto:michael.s.klishin@gmail.com" target="_blank">michael.s.klishin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><br><div class="gmail_quote">2013/9/23 Mike Hadlow <span dir="ltr"><<a href="mailto:mike@suteki.co.uk" target="_blank">mike@suteki.co.uk</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
You shouldn't share a channel between multiple threads, but given that an ACK should be sent on the same channel that the delivery is received mean that one _must_ violate this rule?</blockquote></div><br></div></div>
<div class="gmail_extra">
If you run one consumer per thread, use a new channel for every one of them.<br></div><div><div class="gmail_extra">-- <br>MK<br><br><a href="http://github.com/michaelklishin" target="_blank">http://github.com/michaelklishin</a><br>
<a href="http://twitter.com/michaelklishin" target="_blank">http://twitter.com/michaelklishin</a><br>
</div></div></div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>
_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</blockquote></div><br></div>_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br></blockquote></div><br></div></div></div></div></div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>