[rabbitmq-discuss] Performance degrades with increasing queue depth

[aisha fenton]

Hi,
I'm sure I'm doing something wrong since I can't find reference to
this anywhere else. What I'm seeing is that the performance of
draining a queue gets slower as the queue size increases.

I'm aware of the issue in RabbitMQ 1.6 that means that when it runs
out of physical memory that it's performance degrades because it
starts swapping out. But I'm not anywhere close to running out of
memory yet, and the degradation starts almost immediately and
increases linearly as the queue depth grows.

I am publishing 500mps to an exchange. Each message is about 1KB. I
have a single fanout queue bound to the exchange. A single consumer is
popping messages off the queue.

When the queue is less than 20,000 messages I can pop 300mps off the
queue. When the queue is 200,000 messages the performance drop to
40-80mps.

I'm running RabbitMQ 1.6.0. And nether rabbitmq, the consumer, or the
publisher are using more than 40% CPU.

I assume I shouldn't be seeing this? Any help much appreciated.

Aisha