[rabbitmq-discuss] Performance degrades with increasing queue depth

[Matthias Radestock]

Aisha, Chris,

Chris Duncan wrote:
> 1. In your test_push.rb you have the line -
> 
> /msg_server.exchange(EXCHANGE).publish("This is my msg")/ // The 
> exchange method declares an exchange, so you are doing this every 
> time you publish a message

I've run Aisha's original test_push.rb code through the tracer
(http://www.rabbitmq.com/examples.html#tracer), and can see only *one*
exchange.declare, right at the beginning. So that's not the problem.

Aisha Fenton wrote:
> Code below uses tmm1-amqp instead of Bunny. I get 2000mps using this
> code, draining from the same queue.

As Chris mentioned in his response, this uses a subscription, which is a 
much more efficient way of retrieving messages than basic.get. Also, on 
the publishing side, test_push.rb comes nowhere close to maxing out 
rabbit - I guess ruby is simply too slow for that ;)

Back to your original query though ...

> To isolate the problem, I've reduce what I'm doing down to the
> attached code files. One pushes messages, the other pops them.

I ran the tests and got the following results:

- 20k messages: 500Hz, with beam.smp using 30% of the CPU

- 200k messages: 300Hz, with beam.smp using 45% of the CPU - that kind 
of slowdown is not totally unexpected and probably due to the increased 
cost of gc and the nature of the queue data structure we are using,

- 200k messages: 35Hz, with beam.smp using 95% of the CPU - yes, that's 
right, the same test as the previous one but only 1/9th of the throughput!

So there is definitely something odd going on here.

Digging a little deeper, I think the observed behaviour may have to do 
with Erlang process hibernation. I consistently get the slower result 
when I start consuming after the queue process has gone into 
hibernation, which happens after about one second of idleness. By 
contrast, if I start consuming straight away, without the process 
hibernating first, I consistently get the higher rate.

I have filed a bug to investigate this further.


Regards,

Matthias.