[rabbitmq-discuss] Performance degrades with increasing queue depth

Mon Sep 7 17:32:58 BST 2009

I almost feel as though rabbitmq can't push items to my consumer fast
enough, I had a backed up queue with 20 consumers and spawned another
watching the items get processed and it just seemed so slow, saw numerous
pauses between item processing and I don't believe it is the consumer's
fault.
I am using the Python library for the consumer part.

Suhail

On Mon, Sep 7, 2009 at 8:23 AM, Chuck Remes <cremes.devlist at mac.com> wrote:

>
> On Sep 7, 2009, at 1:07 AM, aisha fenton wrote:
>
> > Hi,
> > I'm sure I'm doing something wrong since I can't find reference to
> > this anywhere else. What I'm seeing is that the performance of
> > draining a queue gets slower as the queue size increases.
> >
> > I'm aware of the issue in RabbitMQ 1.6 that means that when it runs
> > out of physical memory that it's performance degrades because it
> > starts swapping out. But I'm not anywhere close to running out of
> > memory yet, and the degradation starts almost immediately and
> > increases linearly as the queue depth grows.
> >
> > I am publishing 500mps to an exchange. Each message is about 1KB. I
> > have a single fanout queue bound to the exchange. A single consumer is
> > popping messages off the queue.
> >
> > When the queue is less than 20,000 messages I can pop 300mps off the
> > queue. When the queue is 200,000 messages the performance drop to
> > 40-80mps.
> >
> > I'm running RabbitMQ 1.6.0. And nether rabbitmq, the consumer, or the
> > publisher are using more than 40% CPU.
> >
> > I assume I shouldn't be seeing this? Any help much appreciated.
>
> You didn't include any code, but I'm going to take a stab in the dark
> anyway. If you are using the ruby amqp gem you might be doing
> something like this:
>
> def next_message
>   exchange = MQ.fanout 'foo'
>   queue = MQ.queue 'bar'
>
>   queue.bind exchange
>
>   newest_message = queue.pop
> end
>
> In the code above the call to MQ.<whatever> is opening a new channel
> to rabbitmq each time the method is called. You are essentially
> leaking channels all over the place every time you try to pop a new
> message.
>
> If you aren't using the ruby amqp stuff, I still recommend checking
> your code for whatever library you chose. You only need to allocate a
> few channels and reuse them.
>
> cr
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>

-- 
http://mixpanel.com
Blog: http://blog.mixpanel.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090907/3f720c59/attachment.htm 