[rabbitmq-discuss] Channel best practices

[Steve Powell]

(Previous message sent before finalisation -- please ignore it.)

Sorry to butt in, but I wanted to clarify a few things (about the Java client):

There is no ‘rule’ “you shouldn’t share a channel between threads”.  It is (good) advice, though.

It is not true that “basic.deliver will be called on the client library’s internal message loop thread”, handleDelivery is called on a Consumer callback thread.

It is fine to call ack from a Consumer.handleDelivery method, and it is OK to call any other channel method, but it is hard to program multi-thread control of a channel without getting into serious difficulties.

Some channel commands don't wait for a response, and some do.  If you are waiting for a response then your application thread is waiting (blocked). If you issue a second command that wants to wait for a response, the Java Client will block the second thread until the response has returned from the first.  This is thread-safe, and as designed.  However, there is no serialisation of these requests: if many simultaneously issue commands, they may execute in any order.  The state of the channel each command sees is therefore not predictable, and so the application threads need to cope with potential failures and reorderings.

Even commands that don't expect a response can interfere with another thread's commands by changing the state of the channel, so it is quite important that the application threads know what each other are doing.

Issuing acks from many threads runs the risk of a double ack (acking the same message twice), which is a protocol error.  Multi-acks suffer from the same problem.

I refer you to the following sections on the Java Client API documentation page (http://www.rabbitmq.com/api-guide.html):
> Channel thread-safety
> Channel instances are safe for use by multiple threads. Requests into aChannel are serialized, with only one thread being able to run a command on the Channel at a time. Even so, applications should prefer using a Channel per thread instead of sharing the same Channel across multiple threads.
> 
and, in the following section:
> Callbacks to Consumers are dispatched on a thread separate from the thread managed by the Connection. This means that Consumers can safely call blocking methods on the Connection or Channel, such asqueueDeclare, txCommit, basicCancel or basicPublish.
> 
> Each Channel has its own dispatch thread. For the most common use case of one Consumer per Channel, this means Consumers do not hold up other Consumers. If you have multiple Consumers per Channel be aware that a long-running Consumer may hold up dispatch of callbacks to other Consumers on that Channel.
> 
If you are interested….  (background info):

It used to be the case that the consumer callbacks (e.g. handleDelivery) executed on the Connection thread. (This is no longer true—see below.) This meant it was quite dangerous to issue channel commands in a callback method:  the same thread that listened for frames from the server, might block waiting for a response from a channel command.  Deadlock.

The basic.ack command was alright, because there is no response waited for (and any other command which doesn’t have a protocol response is fine, too).  However, many applications tried to issue channel commands (like declareQueue or createChannel) which did expect responses, so they ran into trouble.

To obviate this problem, the QueueingConsumer was created: this essentially does all the necessary things in the callback to receive the message, puts it in a (communicating, thread-safe) Java queue, and returns.  The message can then be obtained from the Java queue by any other thread, including the original application thread.  If channel commands are subsequently issued they cannot block the Connection thread.

Issuing channel commands from more than one thread ran the risk of issuing a command requiring a response (a ‘synchronous’ command) while another thread was already waiting for a response.  This would be an error (a thread-safety error) because the client cannot cope with overlapping synchronous commands on the same channel.  This is prevented: a thread that issues a command expecting a response will block until any outstanding response is received on that channel.

Although the thread-safety issues are (should be—let’s not get complacent) now all fixed, this solution just made a deadlock on the Connection thread more likely if you were trying to code your own Consumer.

In RabbitMQ 2.7.0 and 2.7.1, the Java client was changed to take the callback executions off of the Connection thread altogether. This was for three reasons.  One was to allow channel commands to be issued in handleDelivery (et al.), which, by the way, meant that bespoke Consumers were easier to write—QueueingConsumer is now not (so) necessary.  The second was to recover more gracefully from Consumers that do silly things: throwing exceptions, hanging indefinitely, and so on.  And the third was to prevent Consumers on distinct channels from holding each other up.  (There is now a pool of threads used to execute Consumer callbacks, each channel’s callbacks being executed serially, but distinct channels’ callbacks being allowed to run concurrently.)

It is quite safe to issue an ack (or nack) from another thread.  But you should beware of doing this twice, so multi-threaded applications need to be carefully designed to avoid ‘confusion’ over which thread does what—“channel control fights”.

Channel commands can be issued from multiple threads if you can ensure that the correct channel protocol is observed.  To do this properly requires communication between the threads in the application, which can be non-trivial to get right (I recommend Java Concurrency in Practice, Addison Wesley, Goetz, et al.).

So, the ‘advice’ is not to share channels between application threads, but it is just advice.  It is expected that you might issue (n)acks from the Consumer handleDelivery method, and put other Channel method calls there, too.  No rules are being ‘broken’.

Steve Powell  [Cell: +44-7815-838-558] [RabbitMQ, Pivotal]
“L’enfer, c’est les autres.” Sartre

On 23 Sep 2013, at 15:52, Mike Hadlow <mike at suteki.co.uk> wrote:

> Hi Michael, Tim,
> 
> I've put your comments into a blog post: http://mikehadlow.blogspot.co.uk/2013/09/rabbitmq-amqp-channel-best-practices.html.
> 
> Let me know if there's anything you'd like to add/change/remove.
> 
> Many thanks
> Mike
> 
> 
> On Mon, Sep 23, 2013 at 12:35 PM, Michael Klishin <michael.s.klishin at gmail.com> wrote:
> 
> 2013/9/23 Mike Hadlow <mike at suteki.co.uk>
> You shouldn't share a channel between multiple threads, but given that an ACK should be sent on the same channel that the delivery is received mean that one _must_ violate this rule?
> 
> If you run one consumer per thread, use a new channel for every one of them.
> -- 
> MK
> 
> http://github.com/michaelklishin
> http://twitter.com/michaelklishin
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130924/06dbf5df/attachment.htm>