[rabbitmq-discuss] Slow throughput on consume side using java client API

[scott w]

Hi Matthew --

Thanks for the feedback. Comments below.

cheers,
Scott

On Thu, Nov 26, 2009 at 2:10 AM, Matthew Sackman <matthew at lshift.net> wrote:

> Hi Scott,
>
> I must admit I'm a little confused by your description as it would
> appear somewhat complex to me. However, am I correct in the following?
>

Not quite. Let me clarify.

>
> 1. You have 1 queue, which is being published to by 50 threads.
>

Yes although for the purposes of this discussion the publishing side is not
important. You can just assume the queue is filled up with thousands of
messages.


> 2. You have 25 consumers on that queue.
>

2a. 50 threads calling the equivalent of queue.pop()

2b. Within the queue, I have 25 QueueConsumers and 25 channel (each
QueueConsumer takes in 1 channel in its constructor and there is a 1-1
mapping between QueueConsumer and channel)


> 3. To receive, you randomly select a consumer, and get that consumer to
> consume a message.
>

Right so one of 50 threads will call queue.pop(), and internally picks a
random channel, synchronizes on that channel, and then calls consume on the
QueueConsumer that maps 1-1 with that channel.


> If that's correct, I'm not surprised by your findings. Consumers on a
> queue are round-robin'd by the broker. Thus each of your 25 consumers
> will receive one 25th of the messages. Thus if you randomly select a
> consumer then it may well have nothing waiting on it as messages have
> been sent to the other consumers.


This would definitely be true if there was only a single thread. In the case
where there are 50 threads and 25 queue consumers/channels, then on average
there will be 2 threads working on any individual queue consumer/channel.
When the queue starts to drain out, i.e. is almost empty, the impl as I have
it can be sub optimal since some threads may be asking to work on queue
consumers that are empty when there are other queue consumers that have work
to do (something I need to fix) but when the queue isn't close to draining
out, I don't think there should be the problem you describe. I do see the
read throughput go down as it starts to drain out but the 50 reads / sec I
was describing is before it starts to drain.



> Thus you will have to wait for up to
> 25 publishes before that consumer gets a message.
>
> Why are you using so many threads and consumers?
>

I started with one thread and one consumer and was seeing unacceptable read
throughput (well under 10 reads/sec) so I decided to make the # channels of
and threads parameterizable. After trying various permutations, I found 50
threads and 25 channels to work better than other choices although it wasn't
as scientific as it could be.

The documentation says you can not have multiple threads going against a
single channel so my assumption was that if I need to bump up performance
with multiple threads I would need multiple channels and hence multiple
queue consumers (since it doesn't appear you can have a single queue
consumer going against multiple channels)t. Please correct me if I am wrong.
Or even better if you have some consumer side java code you can share that
you know gives high read throughput, if you wouldn't mind sharing that would
be very much appreciated.

thanks,
Scott
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20091127/d2ca9aea/attachment.htm