[rabbitmq-discuss] Consumer Clients as Tomcat Web Applications and Work Queue Configuration

[Kevin Behr]

Steve,

Thanks for your continued help in the matter, you've done a great job at
answering my questions.  I'm going to take your advice and run both
performance and stress tests, starting with a prefetch count of 1, and then
increasing to 2, and so on.  I think that the tests will help to highlight
the integrity of my pipeline (i.e. bottlenecks, network latency, etc).

Thanks again!

Kevin


On Mon, Feb 18, 2013 at 11:25 AM, Steve Powell <steve at rabbitmq.com> wrote:

> Kevin,
> I'm glad you managed to 'solve' your problem.
>
> The acknowledgement (ACK) is to tell the RabbitMQ Server that the client is
> willing to take responsibility for the message: RabbitMQ will not deliver
> this
> message to anyone again.  If the channel closes/stops/dissappears and a
> message sent to it hasn't been acknowledged, or if a message is rejected
> (with
> requeue)  (NACKed with requeue), then RabbitMQ will attempt to deliver the
> message again.  That may or may not be to the same consumer, channel or
> client.  The prefetch count is a way of limiting the number of messages
> RabbitMQ Server will allow unACKed (and unNACKed) on a channel.
>
> Let's say you have one consumer on one channel on each client, and you have
> two clients, A and B. Let's also say that A and B have prefetch count set
> to 2.
> We'll assume that A and B are served in that order by the RabbitMQ server.
>
> If 5 messages arrive on the queue (m1..5) then the messages will arrive on
> the
> QueueingConsumer client queues on A and B as follows:
>
> A:  [m1, m3]
> B:  [m2, m4]
> and message m5 will stay in the server.
>
> I presume both A and B will see a message and remove it from the
> QueueingConsumer queue:
>
> A:  [m3]  (processing m1)
> B:  [m4]  (processing m2)
>
> When the first message is processed successfully (either m1 on A, or m2 on
> B)
> and the client sends ACK back to the server, then message m5 can be sent to
> the client who ACKed first, let's say it is B:
>
> A:  [m3]  (processing m1)
> B:  [m4, m5]   (not processing a message)
>
> But note that B might have already started processing m4 by this time (it
> more probably would have, since the message m4 is already in the client),
> so it would soon (or already) look like this:
>
> A:  [m3]  (processing m1)
> B:  [m5]  (processing m4)
>
> As messages arrive on the queue in the server, they are sent to either A
> or B
> as they successfully (or otherwise) finish processing the previous
> messages.
>
> Let's say that B finishes first again (ACKnowledging m4), but that  no more
> messages arrive on the queue before then.  We then get into this situation:
>
> A:  [m3]  (processing m1)
> B:  []  (processing m5)
>
> B has an empty queue.
>
> If B again finishes first (a fast and efficient machine, or a lightweight
> message m5?) we could see this situation:
>
> A:  [m3]  (processing m1)
> B:  []  (idle, waiting on internal queue)
>
> A might take a very long time to process message m1, and m3 has to wait
> until
> A has finished (or aborts) before it will be processed, even though B is
> unoccupied and available.
>
> The effects of the prefetch count >1 are therefore three-fold:
>
> 1) The internal client queues never hold more than prefetch-count messages,
>    all excess messages 'back-up' in the server.
>
> 2) Some messages can get 'caught' behind a message that takes a long time
> to
>    process: waiting indefinitely, even though there may be servers ready
> and
>    waiting to process them.
>
> 3) Messages can be sent to a client while it is processing the previous
>    message, so the network transfer time can overlap processing time,
> reducing
>    the time spent idle in the client when there are messages to process.
>
> Notice that there is a (potential) price to pay for the benefit in item 3.
>  If
> you reduce the time spent waiting for network transfer time while idle, you
> run the risk of processing some messages very late indeed.
>
> As soon as you have multiple clients listening to the same queue you cannot
> guarantee the order of processing of messages, whatever you set the
> prefetch
> counts to.
>
> You should be able to see what happens (can happen) with more clients, or
> with
> a larger (or unlimited) prefetch count.
>
> Whether you wish to use a setting like this very much depends upon how
> large
> the network latency is, what the risk of message failures are, and what the
> rate of message arrival might be (it will doubtless vary in any case).
>
> A prefetch count of 2 and a variable number of clients works reasonably
> well:
>
> - when order of message processing is not critical;
> - when message processing is relatively reliable, and not too variable; and
> - when the network transfer time is not too large per message (e.g.
> messages are
>   reasonably small).
>
> Outside of these parameters I recommend you run some careful tests.
>
> In fact, let me recommend you run some careful performance and stress
> tests in
> any case.  It is notoriously difficult to predict where the bottlenecks
> will
> be in any particular configuration.  These bottlenecks will move depending
> upon load, and sometimes when the load changes (up or down).
>
> It may be, with low to medium message rates, that all this fuss makes no
> difference over a prefetch count of 1.  You should check this before
> investing
> in complex (and potentially buggy) code. You should also ensure that you
> really need many processing clients and that you fully understand the
> interactions between the processing that these clients undertake -- it
> is possible that there are mutual blocks or other serialisation effects
> that will have an impact upon the service rates (or even success)
> of your clients.
>
> I hope I have begun to answer your question.  Good luck.
>   Steve Powell
> [*M*: +44-7815-838-558; *H*:+44-1962-775-598]
> *Links: SpringSource <http://www.springsource.org/>** **(**a division of *
> *VMware* <http://www.vmware.com/>*),*
>
> * **Virgo* <http://www.eclipse.org/virgo>*, *
> *RabbitMQ <http://www.rabbitmq.com>.*
> *-----------------------------------------------------------------------*
>   Good design:
>    is innovative, useful, aesthetic;
>    is understandable, unobtrusive, honest;
>    is long-lasting, thorough, environmentally friendly;
>     and *is as little design as possible.*
> *Copyright Dieter Rams, amended March 2003; October 2009; and August 2012*
>
> On 12 Feb 2013, at 19:28, Kevin Behr <behrk2 at gmail.com> wrote:
>
> Regarding your statement above, I had not taken that into consideration.
>  I've coded my clients to  only acknowledge upon successful processing of
> the delivered message.  If one of our many web services or datastore
> connections goes down, then unacknowledged messages will start piling up
> quickly, and the messaging and queuing pipeline will essentially come to a
> halt. I would expect this to be the correct behavior, however, as I don't
> want to throw out messages until someone discovers that a service is down.
>
> As I will be using QueuingConsumer, I'm now wondering *how *I should
> determine *what *to set my prefetch counts to.  Do you have any further
> advice on this?  I'm still a little confused there.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130226/a733857b/attachment.htm>