[rabbitmq-discuss] Channel best practices

Tue Sep 24 15:36:14 BST 2013

Many thanks Steve! A excellent summary. I've updated my blog post. Thanks
especially for clarifying how the threading around consumers works. I'm
using the .NET client. Is it a straight port of the Java client, or are
there significant differences?

Cheers
Mike

On Tue, Sep 24, 2013 at 11:30 AM, Tim Watson <tim at rabbitmq.com> wrote:

> A superlative follow up - thanks for clearing up the details Steve, and
> for the history lesson (which I for one, found very enlightening)!
>
> Cheers,
> Tim
>
> On 24 Sep 2013, at 10:57, Steve Powell wrote:
>
> (Previous message sent before finalisation -- please ignore it.)
>
> Sorry to butt in, but I wanted to clarify a few things (about the Java
> client):
>
> There is *no* ‘rule’ “you shouldn’t share a channel between threads”.  It
> is (good) advice, though.
>
> It is *not* true that “basic.deliver will be called on the client
> library’s internal message loop thread”, handleDelivery is called on a
> Consumer callback thread.
>
> It is fine to call ack from a Consumer.handleDelivery method, and it is
> OK to call any other channel method, *but* it is hard to program
> multi-thread control of a channel without getting into serious difficulties.
>
> Some channel commands don't wait for a response, and some do.  If you are
> waiting for a response then your application thread is waiting (blocked).
> If you issue a second command that wants to wait for a response, the Java
> Client will block the second thread until the response has returned from
> the first.  This is thread-safe, and as designed.  However, there is no
> serialisation of these requests: if many simultaneously issue commands,
> they may execute in any order.  The state of the channel each command sees
> is therefore not predictable, and so the application threads need to cope
> with potential failures and reorderings.
>
> Even commands that don't expect a response can interfere with another
> thread's commands by changing the state of the channel, so it is quite
> important that the application threads know what each other are doing.
>
> Issuing acks from many threads runs the risk of a double ack (acking the
> same message twice), which is a protocol error.  Multi-acks suffer from
> the same problem.
>
> I refer you to the following sections on the Java Client API documentation
> page (http://www.rabbitmq.com/api-guide.html):
>
> Channel thread-safety
>
> Channel instances are safe for use by multiple threads. Requests into a
> Channel are serialized, with only one thread being able to run a command
> on the Channel at a time. Even so, applications should prefer using a
> Channel per thread instead of sharing the same Channel across multiple
> threads.
>
> and, in the following section:
>
> Callbacks to Consumers are dispatched on a thread separate from the
> thread managed by the Connection. This means that Consumers can safely
> call blocking methods on the Connection or Channel, such asqueueDeclare,
> txCommit, basicCancel or basicPublish.
>
> Each Channel has its own dispatch thread. For the most common use case of
> one Consumer per Channel, this means Consumers do not hold up other
> Consumers. If you have multiple Consumers per Channel be aware that a
> long-running Consumer may hold up dispatch of callbacks to other Consumers
> on that Channel.
>
> If you are interested….  (background info):
>
> It *used to be the case* that the consumer callbacks (e.g. handleDelivery)
> executed on the Connection thread. (This is no longer true—see below.)
> This meant it was quite dangerous to issue channel commands in a callback
> method:  the same thread that listened for frames from the server, might
> block waiting for a response from a channel command.  *Deadlock.*
>
> The basic.ack command was alright, because there is no response waited
> for (and any other command which doesn’t have a protocol response is fine,
> too).  However, many applications tried to issue channel commands (like
> declareQueue or createChannel) which *did* expect responses, so they ran
> into trouble.
>
> To obviate this problem, the QueueingConsumer was created: this
> essentially does all the necessary things in the callback to receive the
> message, puts it in a (communicating, thread-safe) Java queue, and returns.
>  The message can then be obtained from the Java queue by any other thread,
> including the original application thread.  If channel commands are
> subsequently issued they cannot block the Connection thread.
>
> Issuing channel commands from more than one thread ran the risk of issuing
> a command requiring a response (a ‘synchronous’ command) while another
> thread was already waiting for a response.  This would be an error (a
> thread-safety error) because the client cannot cope with overlapping
> synchronous commands on the same channel.  This is prevented: a thread that
> issues a command expecting a response will block until any outstanding
> response is received on that channel.
>
> Although the thread-safety issues are (should be—let’s not get complacent)
> now all fixed, this solution just made a deadlock on the Connection thread
> more likely if you were trying to code your own Consumer.
>
> In RabbitMQ 2.7.0 and 2.7.1, the Java client was changed to take the
> callback executions off of the Connection thread altogether. This was for
> three reasons.  *One* was to allow channel commands to be issued in
> handleDelivery (*et al.*), which, by the way, meant that bespoke Consumers
> were easier to write—QueueingConsumer is now not (so) necessary.  The *
> second* was to recover more gracefully from Consumers that do silly
> things: throwing exceptions, hanging indefinitely, and so on.  And the *
> third* was to prevent Consumers on *distinct* channels from holding each
> other up.  (There is now a pool of threads used to execute Consumer callbacks,
> each channel’s callbacks being executed serially, but distinct channels’
> callbacks being allowed to run concurrently.)
>
> It is quite safe to issue an ack (or nack) from another thread.  But you
> should beware of doing this twice, so multi-threaded applications need to
> be carefully designed to avoid ‘confusion’ over which thread does
> what—“channel control fights”.
>
> Channel commands can be issued from multiple threads if you can ensure
> that the correct channel protocol is observed.  To do this properly
> requires communication between the threads in the application, which can be
> non-trivial to get right (I recommend *Java Concurrency in Practice*,
> Addison Wesley, Goetz, et al.).
>
> So, the ‘*advice*’ is *not* to share channels between application
> threads, but it is *just advice*.  It is expected that you might issue (n)
> acks from the Consumer handleDelivery method, and put other Channel method
> calls there, too.  No rules are being ‘broken’.
>
> Steve Powell  [*Cell*: +44-7815-838-558] [RabbitMQ<http://www.rabbitmq.com/>
> ,* *Pivotal <http://gopivotal.com/>]
> *“L’enfer, c’est les autres.” *Sartre
>
> On 23 Sep 2013, at 15:52, Mike Hadlow <mike at suteki.co.uk> wrote:
>
> Hi Michael, Tim,
>
> I've put your comments into a blog post:
> http://mikehadlow.blogspot.co.uk/2013/09/rabbitmq-amqp-channel-best-practices.html
> .
>
> Let me know if there's anything you'd like to add/change/remove.
>
> Many thanks
> Mike
>
>
> On Mon, Sep 23, 2013 at 12:35 PM, Michael Klishin <
> michael.s.klishin at gmail.com> wrote:
>
>>
>> 2013/9/23 Mike Hadlow <mike at suteki.co.uk>
>>
>>> You shouldn't share a channel between multiple threads, but given that
>>> an ACK should be sent on the same channel that the delivery is received
>>> mean that one _must_ violate this rule?
>>
>>
>>  If you run one consumer per thread, use a new channel for every one of
>> them.
>> --
>> MK
>>
>> http://github.com/michaelklishin
>> http://twitter.com/michaelklishin
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130924/5f16a1ec/attachment.htm>