[rabbitmq-discuss] Performance degrades with increasing queue depth

[Chris Duncan]

Hi Aisha,

Hope you don't mind me butting in here.

2009/9/7 Aisha Fenton <aisha.fenton at gmail.com>

>  Hi Chunk.
> I don't know Suhail. And his problem may be different from mine.
>
>  To isolate the problem, I've reduce what I'm doing down to the attached
> code files. One pushes messages, the other pops them.
>
> I see the problem even when only test_push.rb is running (as long as the
> queue is sufficiently pre-populated). Nothing else is using rabbitmq when
> I'm doing the test.
>
> I'm getting the same problem on several different machines here -- my local
> MacOSX box and a server class Debian box.
>
>
>   exchange - durable or not?
>
> not durable
>
>  exchange - type? (Looks like "fanout" from the original post)
>
> fanout
>
>  number of bindings per exchange
>
> 1
>
>  queue  - durable?
>
> not durable
>
>  publishing settings - require ack? persistent? immediate?
>
> no ack, non persistent, and non immediate.
>
>  message size? 1K according to the original post
>
> < 1K
>
>
>
> *test_push.rb *
> *---*
>
>  *require "rubygems"*
> *require "bunny"*
> *
> *
> *EXCHANGE = "raw_telemetry"*
> *QUEUE = "process_telemetry"*
>  *
> *
> *count ||= 0*
> *prev_time ||= Time.now*
>  *msg_server = Bunny.new(:spec => '08')*
> *msg_server.start*
> *
> *
> *msg_server.exchange(EXCHANGE, :type  => :fanout)*
> *msg_server.queue(QUEUE).bind(EXCHANGE)*
> *
> *
> *loop do*
> *
> *
> *  msg_server.exchange(EXCHANGE).publish("This is my msg")*
>  *  *
> *  count += 1*
> *  if count > 100*
> *    t = Time.now*
> *    puts("msgs pushes per sec: #{count / (t - prev_time)}")*
>  *    prev_time = t*
> *    count = 0*
> *  end*
> *    *
> *end*
> *
> *
> * test_pop.rb
> ---
>
> *
> * require "rubygems"
> require "bunny"
>
> @msg_server = Bunny.new(:spec => '08')
> @msg_server.start
>
> def start
>   count ||= 0
>   prev_time ||= Time.now
>   queue = @msg_server.queue("process_telemetry")
>
>    loop do
>     result = queue.pop
>     next if result == :queue_empty
>
>     count += 1
>     if count > 100
>       t = Time.now
>       puts("msgs pops per sec: #{count / (t - prev_time)}")
>       prev_time = t
>       count = 0
>     end
>   end
>
> end
>
> begin
>   start()
> ensure
>   puts "stopping"
>   @msg_server.stop
> end
> *
>

I have some observations -

1. In your test_push.rb you have the line -

*msg_server.exchange(EXCHANGE).publish("This is my msg")*
**
The exchange method declares an exchange, so you are doing this every time
you publish a message which will slow things down. Declare the exchange
using a variable outside the loop like this -

*my_exchange = msg_server.exchange(EXCHANGE, :type  => :fanout)*
**
Then use the variable inside the loop -
**
*my_exchange.publish("This is my msg")*
**
I think Paolo has highlighted the same thing.

2. In your test_pop.rb you don't need to specify *:spec => '08'.* Bunny.new
defaults to the AMQP 0-8 spec version.

Also, do you mean to have -

*next if result == :queue_empty*
**
How are you breaking out of the loop? Might you want to use 'break' instead
of 'next'?

3. A Bunny instance only ever uses 2 channels unless you explicitly create
more. Since you don't seem to be creating more channels I don't think that's
where your issue lies.

-- 
Regards

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090908/ec5effd0/attachment.htm