[rabbitmq-discuss] Consumer Clients as Tomcat Web Applications and Work Queue Configuration
Kevin Behr
behrk2 at gmail.com
Tue Feb 12 19:28:51 GMT 2013
Steve,
Thanks very much for your response. I was able to re-implement the while
loop without having Tomcat bug-out upon server stop. I achieved this by
catching the the InterruptedException in a Thread:
while (!Thread.currentThread().isInterrupted()) {
...
channel.basicQos(1);
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume(ReferenceData.CONSUME_QUEUE_NAME, false,
consumer)
try {
QueueingConsumer.Delivery delivery =
consumer.nextDelivery();
...
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
break;
}
}
I find your comments about the polling interesting. For now, though, my
requirements would seem to favor as little network latency as possible, so
I'm going to go with the original push/round robin model.
I am more confident in deploying the consumers and producers as web
applications, and your answer has helped to confirm that. I am finding
benefit in manageability and portability and my ultimate goal, like you
said, is a sophisticated web-based control mechanism. I want to have
real-time control of my clients, and I want it to be user friendly. I've
started looking into JMX and MBeans to achieve such a solution.
*If you impose a prefetch
limit of 1 for each client you run, then the rabbit server will dispatch
messages to clients that have no unacknowledged messages (by the way,
you should explicitly acknowledge messages when using non-zero
basicQos()).*
*
*
Regarding your statement above, I had not taken that into consideration.
I've coded my clients to only acknowledge upon successful processing of
the delivered message. If one of our many web services or datastore
connections goes down, then unacknowledged messages will start piling up
quickly, and the messaging and queuing pipeline will essentially come to a
halt. I would expect this to be the correct behavior, however, as I don't
want to throw out messages until someone discovers that a service is down.
As I will be using QueuingConsumer, I'm now wondering *how *I should
determine *what *to set my prefetch counts to. Do you have any further
advice on this? I'm still a little confused there.
Thanks again,
Kevin Behr
On Mon, Feb 11, 2013 at 6:33 AM, Steve Powell <steve at rabbitmq.com> wrote:
> Hi Kevin,
>
> Sorry for the delayed response.
>
> First, an observation about your original set-up:
>
> > while (true) {
> > QueueingConsumer.Delivery delivery = consumer.nextDelivery();
> > String message = new String(delivery.getBody());
> > System.out.println(" [x] Received '" + message + "'");
> > }
>
>
> I assume that there are no Qos (prefetch) limits imposed on the channels
> these consumers use, and also I note that you are using the
> QueueingConsumer. I think this means that all of the consumers are able
> to receive the messages all the time, and so all messages will get
> distributed in round-robin fashion, until there are no messages left to
> deliver.
>
> In your 'workaround' solution, you are indeed pulling, not being pushed,
> and here all of the consumers will get a look-in: there is likely to be
> balancing, but not round-robin. This is based instead upon consumer
> request, so if one of them gets bogged down in work (initiated by the
> previous message, for example) then the other consumers can poll and
> take the next message, and, crucially, other messages won't be taken by
> this consumer while it is busy. Although this isn't 'round-robin'
> servicing, it is arguably better: the messages are not 'blindly' sent to
> consumers that are busy.
>
> The drawbacks are, as you note, an overhead of polling in the consumers,
> and a larger latency on the network -- each get involves a round-trip to
> the server, even when there is nothing to get.
>
> Now to your questions:
>
> > • Is it a bad idea to deploy consumers as Tomcat web applications? Are
> > there any advantages or disadvantages as compared to the alternative of
> > running system level Java clients?
>
> I see no reason why, apart from the overhead implied by running a full
> web application, there should be any particular disadvantages of running
> a Tomcat web application that does messaging as its main work. In fact,
> I suspect it is easier to control (start/stop) than using a
> free-standing Java client: you can potentially build a sophisticated
> web-based control mechanism. However, I'm not best placed to answer
> this.
>
> > • In the case of work queues, am I better off retrieving messages that
> > are pushed down to my clients, and setting a prefetchCount
> > (channel.basicQos(1)) to ensure that I only receive one message at a
> > time?
>
> This is the crux of the matter, as I see it. If you impose a prefetch
> limit of 1 for each client you run, then the rabbit server will dispatch
> messages to clients that have no unacknowledged messages (by the way,
> you should explicitly acknowledge messages when using non-zero
> basicQos()). Under these circumstances, you can achieve similar
> balancing as in the polling case, but without the polling. It will *not*
> be strictly round-robin after the first 'round' because the first
> consumer need not have acknowledged the previous message yet, and
> messages will be delivered to whatever consumers are ready for them --
> albeit in round-robin priority.
>
> The tutorial pages describe this pretty well (see [1]).
>
> There are two points to note:
>
> If you are using QueueingConsumer, then each message will be placed in a
> client Java queue, and nextDelivery() then unblocks with the message
> when it arrives. With explicit acknowledgements this means that one
> message at a time is transferred to the Java queue. You could increase
> the prefetchCount to 2, say, and then up to two messages would be placed
> in the queue but you would still process them serially through the
> nextDelivery() call. When one consumer gets two messages, then no other
> consumer will get these -- so if a message can take a long time before
> it is acknowledged, the message behind it can get delayed, even though
> there may be other consumers which are idle. Other messages can
> 'overtake' and get processed before it.
>
> If you want to avoid the QueueingConsumer internal Java queue, you could
> write your own Consumer implementation (base it on DefaultConsumer) but
> you should be aware that the handleDelivery() methods are driven
> asynchronously, but serially. You shouldn't perform long pieces of work
> on these threads. It would be wise to pass the message to some
> serialised worker thread, especially if you have a prefetchCount larger
> than one. This will probably then require a queue of your own anyway, so
> the advantages to you are slim.
>
> I hope this helps. Please let us know how you get on.
>
> Steve Powell
>
> [1] http://www.rabbitmq.com/tutorials/tutorial-two-java.html
>
> [M: +44-7815-838-558; H:+44-1962-775-598]
> SpringSource (a division of VMware), Virgo, RabbitMQ.
> -----------------------------------------------------------------------
> Good design:
> is innovative, useful, aesthetic;
> is understandable, unobtrusive, honest;
> is long-lasting, thorough, environmentally friendly;
> and is as little design as possible.
> Copyright Dieter Rams, amended March 2003; October 2009; and August 2012
>
> On 31 Jan 2013, at 01:33, Kevin Behr <behrk2 at gmail.com> wrote:
> > Hello everyone,
> >
> > I have five RabbitMQ consumer clients that are written and deployed as
> Tomcat web applications. My original intention was to consume messages in
> the typical fashion:
> …(elided)
> > I would appreciate any advice or suggestions.
> >
> > Thanks!
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130212/ba95f3d3/attachment.htm>
More information about the rabbitmq-discuss
mailing list