[rabbitmq-discuss] Performance degrades with increasing queue depth

[Aisha Fenton]

Code below.

Chuck, I think you're onto something. Something that I noticed while  
putting together the test code is that it goes 300mps the first time I  
run it. Any subsequent runs drop to 80mps. Only a rabbitmq restart  
gets it back to 300mps, and then only for the first run. Hmmm sounds  
like a connection leak...

I can't see anywhere in the code where I'm leaking connections (?) but  
maybe Bunny (the AMQP Ruby adapter I'm using) is?

----
require "rubygems"
require "bunny"

@msg_server = Bunny.new(:spec => '08')
@msg_server.start

def start
   count ||= 0
   prev_time ||= Time.now
   queue = @msg_server.queue("process_telemetry")

    # main loop
   loop do
     result = queue.pop
     next if result == :queue_empty

     count += 1
     if count > 100
       t = Time.now
       puts("msgs pops per sec: #{count / (t - prev_time)}")
       prev_time = t
       count = 0
     end
   end

end

begin
   start()
ensure
   puts "stopping"
   @msg_server.stop
end


On 8/09/2009, at 3:23 AM, Chuck Remes wrote:

>
> On Sep 7, 2009, at 1:07 AM, aisha fenton wrote:
>
>> Hi,
>> I'm sure I'm doing something wrong since I can't find reference to
>> this anywhere else. What I'm seeing is that the performance of
>> draining a queue gets slower as the queue size increases.
>>
>> I'm aware of the issue in RabbitMQ 1.6 that means that when it runs
>> out of physical memory that it's performance degrades because it
>> starts swapping out. But I'm not anywhere close to running out of
>> memory yet, and the degradation starts almost immediately and
>> increases linearly as the queue depth grows.
>>
>> I am publishing 500mps to an exchange. Each message is about 1KB. I
>> have a single fanout queue bound to the exchange. A single consumer  
>> is
>> popping messages off the queue.
>>
>> When the queue is less than 20,000 messages I can pop 300mps off the
>> queue. When the queue is 200,000 messages the performance drop to
>> 40-80mps.
>>
>> I'm running RabbitMQ 1.6.0. And nether rabbitmq, the consumer, or the
>> publisher are using more than 40% CPU.
>>
>> I assume I shouldn't be seeing this? Any help much appreciated.
>
> You didn't include any code, but I'm going to take a stab in the  
> dark anyway. If you are using the ruby amqp gem you might be doing  
> something like this:
>
> def next_message
>  exchange = MQ.fanout 'foo'
>  queue = MQ.queue 'bar'
>
>  queue.bind exchange
>
>  newest_message = queue.pop
> end
>
> In the code above the call to MQ.<whatever> is opening a new channel  
> to rabbitmq each time the method is called. You are essentially  
> leaking channels all over the place every time you try to pop a new  
> message.
>
> If you aren't using the ruby amqp stuff, I still recommend checking  
> your code for whatever library you chose. You only need to allocate  
> a few channels and reuse them.
>
> cr
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090908/a5ad2431/attachment.htm