[rabbitmq-discuss] Performance degrades with increasing queue depth

[Chuck Remes]

Are you running the same code as Aisha? Are you the same person or  
work together? I'm getting a bit confused in this thread.

And, someone needs to start showing some code. I'm able to push  
hundreds of messages per second and never see a drop off in delivery  
performance regardless of the queue size (until I run out of memory).  
I don't think it is fair to blame rabbit until we can see your code  
and understand what you are doing.

Also, please specify the following things:

exchange - durable or not?
exchange - type? (Looks like "fanout" from the original post)
number of bindings per exchange
queue  - durable?
publishing settings - require ack? persistent? immediate?
message size? 1K according to the original post

Is your test machine paging or swapping? What is the 'top' output for  
your process and for rabbit at various points of the run cycle? Is it  
lower or higher when you are getting 300 msg/s than when you are  
getting 80 msg/s?

What is your subscriber doing with the data it receives? If you make  
that process a "no op" how many messages per second can you handle?  
Does it ever drop off related to queue size? Why are you "popping"  
messages from the queue isn't of having them pushed automatically?

This thread has been really frustrating because there has NOT been  
very much information shared. All I see is "it's slow." Give us more  
information.

cr


On Sep 7, 2009, at 11:34 AM, Suhail Doshi wrote:

> 2009-09-07 16:33:10,822 INFO [id: 95705166] Processing complete,  
> acknowledged.
> 2009-09-07 16:33:10,826 INFO [id: 95705166] Received queue item,  
> processing...
> 2009-09-07 16:33:10,878 INFO [id: 95705166] Processing complete,  
> acknowledged.
> 2009-09-07 16:33:10,882 INFO [id: 95705166] Received queue item,  
> processing...
> 2009-09-07 16:33:12,839 INFO [id: 95705166] Processing complete,  
> acknowledged.
> 2009-09-07 16:33:12,839 INFO [id: 95705166] Received queue item,  
> processing...
> 2009-09-07 16:33:13,531 INFO [id: 95705166] Processing complete,  
> acknowledged.
> 2009-09-07 16:33:13,531 INFO [id: 95705166] Received queue item,  
> processing...
>
> If you see there's like a pause for a second between 10 and 12, an  
> item gets processed pretty fast as you can see. Odd isn't it?
>
> Suhail
>
> On Mon, Sep 7, 2009 at 9:32 AM, Suhail Doshi  
> <digitalwarfare at gmail.com> wrote:
> I almost feel as though rabbitmq can't push items to my consumer  
> fast enough, I had a backed up queue with 20 consumers and spawned  
> another watching the items get processed and it just seemed so slow,  
> saw numerous pauses between item processing and I don't believe it  
> is the consumer's fault.
>
> I am using the Python library for the consumer part.
>
> Suhail
>
>
> On Mon, Sep 7, 2009 at 8:23 AM, Chuck Remes <cremes.devlist at mac.com>  
> wrote:
>
> On Sep 7, 2009, at 1:07 AM, aisha fenton wrote:
>
> > Hi,
> > I'm sure I'm doing something wrong since I can't find reference to
> > this anywhere else. What I'm seeing is that the performance of
> > draining a queue gets slower as the queue size increases.
> >
> > I'm aware of the issue in RabbitMQ 1.6 that means that when it runs
> > out of physical memory that it's performance degrades because it
> > starts swapping out. But I'm not anywhere close to running out of
> > memory yet, and the degradation starts almost immediately and
> > increases linearly as the queue depth grows.
> >
> > I am publishing 500mps to an exchange. Each message is about 1KB. I
> > have a single fanout queue bound to the exchange. A single  
> consumer is
> > popping messages off the queue.
> >
> > When the queue is less than 20,000 messages I can pop 300mps off the
> > queue. When the queue is 200,000 messages the performance drop to
> > 40-80mps.
> >
> > I'm running RabbitMQ 1.6.0. And nether rabbitmq, the consumer, or  
> the
> > publisher are using more than 40% CPU.
> >
> > I assume I shouldn't be seeing this? Any help much appreciated.
>
> You didn't include any code, but I'm going to take a stab in the dark
> anyway. If you are using the ruby amqp gem you might be doing
> something like this:
>
> def next_message
>   exchange = MQ.fanout 'foo'
>   queue = MQ.queue 'bar'
>
>   queue.bind exchange
>
>   newest_message = queue.pop
> end
>
> In the code above the call to MQ.<whatever> is opening a new channel
> to rabbitmq each time the method is called. You are essentially
> leaking channels all over the place every time you try to pop a new
> message.
>
> If you aren't using the ruby amqp stuff, I still recommend checking
> your code for whatever library you chose. You only need to allocate a
> few channels and reuse them.
>
> cr
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> -- 
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
>
>
> -- 
> http://mixpanel.com
> Blog: http://blog.mixpanel.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090907/1f84fdf7/attachment.htm