[rabbitmq-discuss] Slow throughput on consume side using java client API

[scott w]

Just to follow up on my earlier email, I have discovered the culprit. I had
been running the rabbitmq server on EC2 and my performance tests on my
laptop. When I switch to also running the rabbitmq server on my laptop the
read throughput looks great. I'm not sure why the read throughput (and not
the write throughput) slows down so much when running on EC2 but at least
it's good the problem is not coming from rabbitmq.

thanks,
Scott

On Fri, Nov 27, 2009 at 8:18 AM, scott w <scottblanc at gmail.com> wrote:

> Hi Matthew --
>
> Thanks for the feedback. Comments below.
>
> cheers,
> Scott
>
> On Thu, Nov 26, 2009 at 2:10 AM, Matthew Sackman <matthew at lshift.net>wrote:
>
>> Hi Scott,
>>
>> I must admit I'm a little confused by your description as it would
>> appear somewhat complex to me. However, am I correct in the following?
>>
>
> Not quite. Let me clarify.
>
>>
>> 1. You have 1 queue, which is being published to by 50 threads.
>>
>
> Yes although for the purposes of this discussion the publishing side is not
> important. You can just assume the queue is filled up with thousands of
> messages.
>
>
>> 2. You have 25 consumers on that queue.
>>
>
> 2a. 50 threads calling the equivalent of queue.pop()
>
> 2b. Within the queue, I have 25 QueueConsumers and 25 channel (each
> QueueConsumer takes in 1 channel in its constructor and there is a 1-1
> mapping between QueueConsumer and channel)
>
>
>> 3. To receive, you randomly select a consumer, and get that consumer to
>> consume a message.
>>
>
> Right so one of 50 threads will call queue.pop(), and internally picks a
> random channel, synchronizes on that channel, and then calls consume on the
> QueueConsumer that maps 1-1 with that channel.
>
>
>> If that's correct, I'm not surprised by your findings. Consumers on a
>> queue are round-robin'd by the broker. Thus each of your 25 consumers
>> will receive one 25th of the messages. Thus if you randomly select a
>> consumer then it may well have nothing waiting on it as messages have
>> been sent to the other consumers.
>
>
> This would definitely be true if there was only a single thread. In the
> case where there are 50 threads and 25 queue consumers/channels, then on
> average there will be 2 threads working on any individual queue
> consumer/channel. When the queue starts to drain out, i.e. is almost empty,
> the impl as I have it can be sub optimal since some threads may be asking to
> work on queue consumers that are empty when there are other queue consumers
> that have work to do (something I need to fix) but when the queue isn't
> close to draining out, I don't think there should be the problem you
> describe. I do see the read throughput go down as it starts to drain out but
> the 50 reads / sec I was describing is before it starts to drain.
>
>
>
>> Thus you will have to wait for up to
>> 25 publishes before that consumer gets a message.
>>
>> Why are you using so many threads and consumers?
>>
>
> I started with one thread and one consumer and was seeing unacceptable read
> throughput (well under 10 reads/sec) so I decided to make the # channels of
> and threads parameterizable. After trying various permutations, I found 50
> threads and 25 channels to work better than other choices although it wasn't
> as scientific as it could be.
>
> The documentation says you can not have multiple threads going against a
> single channel so my assumption was that if I need to bump up performance
> with multiple threads I would need multiple channels and hence multiple
> queue consumers (since it doesn't appear you can have a single queue
> consumer going against multiple channels)t. Please correct me if I am wrong.
> Or even better if you have some consumer side java code you can share that
> you know gives high read throughput, if you wouldn't mind sharing that would
> be very much appreciated.
>
> thanks,
> Scott
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20091127/116fe9f7/attachment.htm