[rabbitmq-discuss] Performance degrades with increasing queue depth
digitalwarfare at gmail.com
Mon Sep 7 17:32:58 BST 2009
I almost feel as though rabbitmq can't push items to my consumer fast
enough, I had a backed up queue with 20 consumers and spawned another
watching the items get processed and it just seemed so slow, saw numerous
pauses between item processing and I don't believe it is the consumer's
I am using the Python library for the consumer part.
On Mon, Sep 7, 2009 at 8:23 AM, Chuck Remes <cremes.devlist at mac.com> wrote:
> On Sep 7, 2009, at 1:07 AM, aisha fenton wrote:
> > Hi,
> > I'm sure I'm doing something wrong since I can't find reference to
> > this anywhere else. What I'm seeing is that the performance of
> > draining a queue gets slower as the queue size increases.
> > I'm aware of the issue in RabbitMQ 1.6 that means that when it runs
> > out of physical memory that it's performance degrades because it
> > starts swapping out. But I'm not anywhere close to running out of
> > memory yet, and the degradation starts almost immediately and
> > increases linearly as the queue depth grows.
> > I am publishing 500mps to an exchange. Each message is about 1KB. I
> > have a single fanout queue bound to the exchange. A single consumer is
> > popping messages off the queue.
> > When the queue is less than 20,000 messages I can pop 300mps off the
> > queue. When the queue is 200,000 messages the performance drop to
> > 40-80mps.
> > I'm running RabbitMQ 1.6.0. And nether rabbitmq, the consumer, or the
> > publisher are using more than 40% CPU.
> > I assume I shouldn't be seeing this? Any help much appreciated.
> You didn't include any code, but I'm going to take a stab in the dark
> anyway. If you are using the ruby amqp gem you might be doing
> something like this:
> def next_message
> exchange = MQ.fanout 'foo'
> queue = MQ.queue 'bar'
> queue.bind exchange
> newest_message = queue.pop
> In the code above the call to MQ.<whatever> is opening a new channel
> to rabbitmq each time the method is called. You are essentially
> leaking channels all over the place every time you try to pop a new
> If you aren't using the ruby amqp stuff, I still recommend checking
> your code for whatever library you chose. You only need to allocate a
> few channels and reuse them.
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss