Just to follow up on my earlier email, I have discovered the culprit. I had been running the rabbitmq server on EC2 and my performance tests on my laptop. When I switch to also running the rabbitmq server on my laptop the read throughput looks great. I'm not sure why the read throughput (and not the write throughput) slows down so much when running on EC2 but at least it's good the problem is not coming from rabbitmq.<br>
<br>thanks,<br>Scott<br><br><div class="gmail_quote">On Fri, Nov 27, 2009 at 8:18 AM, scott w <span dir="ltr"><<a href="mailto:scottblanc@gmail.com">scottblanc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Matthew --<br><br>Thanks for the feedback. Comments below.<br><br>cheers,<br>Scott<br><br><div class="gmail_quote"><div class="im">On Thu, Nov 26, 2009 at 2:10 AM, Matthew Sackman <span dir="ltr"><<a href="mailto:matthew@lshift.net" target="_blank">matthew@lshift.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi Scott,<br>
<br>
I must admit I'm a little confused by your description as it would<br>
appear somewhat complex to me. However, am I correct in the following?<br></blockquote></div><div><br>Not quite. Let me clarify. <br></div><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
1. You have 1 queue, which is being published to by 50 threads.<br></blockquote></div><div><br>Yes although for the purposes of this discussion the publishing side is not important. You can just assume the queue is filled up with thousands of messages.<br>
<br></div><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
2. You have 25 consumers on that queue.<br></blockquote></div><div><br>2a. 50 threads calling the equivalent of queue.pop()<br><br>2b. Within the queue, I have 25 QueueConsumers and 25 channel (each QueueConsumer takes in 1 channel in its constructor and there is a 1-1 mapping between QueueConsumer and channel)<br>
<br></div><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
3. To receive, you randomly select a consumer, and get that consumer to<br>
consume a message.<br></blockquote></div><div><br>Right so one of 50 threads will call queue.pop(), and internally picks a random channel, synchronizes on that channel, and then calls consume on the QueueConsumer that maps 1-1 with that channel.<br>
<br></div><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
If that's correct, I'm not surprised by your findings. Consumers on a<br>
queue are round-robin'd by the broker. Thus each of your 25 consumers<br>
will receive one 25th of the messages. Thus if you randomly select a<br>
consumer then it may well have nothing waiting on it as messages have<br>
been sent to the other consumers. </blockquote></div><div><br>This would definitely be true if there was only a single thread. In the case where there are 50 threads and 25 queue consumers/channels, then on average there will be 2 threads working on any individual queue consumer/channel. When the queue starts to drain out, i.e. is almost empty, the impl as I have it can be sub optimal since some threads may be asking to work on queue consumers that are empty when there are other queue consumers that have work to do (something I need to fix) but when the queue isn't close to draining out, I don't think there should be the problem you describe. I do see the read throughput go down as it starts to drain out but the 50 reads / sec I was describing is before it starts to drain.<br>
<br> </div><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Thus you will have to wait for up to<br>
25 publishes before that consumer gets a message.<br>
<br>
Why are you using so many threads and consumers?<br></blockquote></div><div><br>I started with one thread and one consumer and was seeing unacceptable read throughput (well under 10 reads/sec) so I decided to make the # channels of and threads parameterizable. After trying various permutations, I found 50 threads and 25 channels to work better than other choices although it wasn't as scientific as it could be.<br>
<br>The documentation says you can not have multiple threads going against a single channel so my assumption was that if I need to bump up performance with multiple threads I would need multiple channels and hence multiple queue consumers (since it doesn't appear you can have a single queue consumer going against multiple channels)t. Please correct me if I am wrong. Or even better if you have some consumer side java code you can share that you know gives high read throughput, if you wouldn't mind sharing that would be very much appreciated.<br>
<br>thanks,<br>Scott<br><br><br></div></div>
</blockquote></div><br>