[rabbitmq-discuss] Performance degrades with increasing queue depth
cremes.devlist at mac.com
Mon Sep 7 16:23:46 BST 2009
On Sep 7, 2009, at 1:07 AM, aisha fenton wrote:
> I'm sure I'm doing something wrong since I can't find reference to
> this anywhere else. What I'm seeing is that the performance of
> draining a queue gets slower as the queue size increases.
> I'm aware of the issue in RabbitMQ 1.6 that means that when it runs
> out of physical memory that it's performance degrades because it
> starts swapping out. But I'm not anywhere close to running out of
> memory yet, and the degradation starts almost immediately and
> increases linearly as the queue depth grows.
> I am publishing 500mps to an exchange. Each message is about 1KB. I
> have a single fanout queue bound to the exchange. A single consumer is
> popping messages off the queue.
> When the queue is less than 20,000 messages I can pop 300mps off the
> queue. When the queue is 200,000 messages the performance drop to
> I'm running RabbitMQ 1.6.0. And nether rabbitmq, the consumer, or the
> publisher are using more than 40% CPU.
> I assume I shouldn't be seeing this? Any help much appreciated.
You didn't include any code, but I'm going to take a stab in the dark
anyway. If you are using the ruby amqp gem you might be doing
something like this:
exchange = MQ.fanout 'foo'
queue = MQ.queue 'bar'
newest_message = queue.pop
In the code above the call to MQ.<whatever> is opening a new channel
to rabbitmq each time the method is called. You are essentially
leaking channels all over the place every time you try to pop a new
If you aren't using the ruby amqp stuff, I still recommend checking
your code for whatever library you chose. You only need to allocate a
few channels and reuse them.
More information about the rabbitmq-discuss