[rabbitmq-discuss] Performance degrades with increasing queue depth

[Chuck Remes]

On Sep 7, 2009, at 1:07 AM, aisha fenton wrote:

> Hi,
> I'm sure I'm doing something wrong since I can't find reference to
> this anywhere else. What I'm seeing is that the performance of
> draining a queue gets slower as the queue size increases.
>
> I'm aware of the issue in RabbitMQ 1.6 that means that when it runs
> out of physical memory that it's performance degrades because it
> starts swapping out. But I'm not anywhere close to running out of
> memory yet, and the degradation starts almost immediately and
> increases linearly as the queue depth grows.
>
> I am publishing 500mps to an exchange. Each message is about 1KB. I
> have a single fanout queue bound to the exchange. A single consumer is
> popping messages off the queue.
>
> When the queue is less than 20,000 messages I can pop 300mps off the
> queue. When the queue is 200,000 messages the performance drop to
> 40-80mps.
>
> I'm running RabbitMQ 1.6.0. And nether rabbitmq, the consumer, or the
> publisher are using more than 40% CPU.
>
> I assume I shouldn't be seeing this? Any help much appreciated.

You didn't include any code, but I'm going to take a stab in the dark  
anyway. If you are using the ruby amqp gem you might be doing  
something like this:

def next_message
   exchange = MQ.fanout 'foo'
   queue = MQ.queue 'bar'

   queue.bind exchange

   newest_message = queue.pop
end

In the code above the call to MQ.<whatever> is opening a new channel  
to rabbitmq each time the method is called. You are essentially  
leaking channels all over the place every time you try to pop a new  
message.

If you aren't using the ruby amqp stuff, I still recommend checking  
your code for whatever library you chose. You only need to allocate a  
few channels and reuse them.

cr