[rabbitmq-discuss] Performance degrades with increasing queue depth

[Suhail Doshi]

2009-09-07 16:33:10,822 INFO [id: 95705166] Processing complete,
acknowledged.
2009-09-07 16:33:10,826 INFO [id: 95705166] Received queue item,
processing...
2009-09-07 16:33:10,878 INFO [id: 95705166] Processing complete,
acknowledged.
2009-09-07 16:33:10,882 INFO [id: 95705166] Received queue item,
processing...
2009-09-07 16:33:12,839 INFO [id: 95705166] Processing complete,
acknowledged.
2009-09-07 16:33:12,839 INFO [id: 95705166] Received queue item,
processing...
2009-09-07 16:33:13,531 INFO [id: 95705166] Processing complete,
acknowledged.
2009-09-07 16:33:13,531 INFO [id: 95705166] Received queue item,
processing...

If you see there's like a pause for a second between 10 and 12, an item gets
processed pretty fast as you can see. Odd isn't it?

Suhail

On Mon, Sep 7, 2009 at 9:32 AM, Suhail Doshi <digitalwarfare at gmail.com>wrote:

> I almost feel as though rabbitmq can't push items to my consumer fast
> enough, I had a backed up queue with 20 consumers and spawned another
> watching the items get processed and it just seemed so slow, saw numerous
> pauses between item processing and I don't believe it is the consumer's
> fault.
> I am using the Python library for the consumer part.
>
> Suhail
>
>
> On Mon, Sep 7, 2009 at 8:23 AM, Chuck Remes <cremes.devlist at mac.com>wrote:
>
>>
>> On Sep 7, 2009, at 1:07 AM, aisha fenton wrote:
>>
>> > Hi,
>> > I'm sure I'm doing something wrong since I can't find reference to
>> > this anywhere else. What I'm seeing is that the performance of
>> > draining a queue gets slower as the queue size increases.
>> >
>> > I'm aware of the issue in RabbitMQ 1.6 that means that when it runs
>> > out of physical memory that it's performance degrades because it
>> > starts swapping out. But I'm not anywhere close to running out of
>> > memory yet, and the degradation starts almost immediately and
>> > increases linearly as the queue depth grows.
>> >
>> > I am publishing 500mps to an exchange. Each message is about 1KB. I
>> > have a single fanout queue bound to the exchange. A single consumer is
>> > popping messages off the queue.
>> >
>> > When the queue is less than 20,000 messages I can pop 300mps off the
>> > queue. When the queue is 200,000 messages the performance drop to
>> > 40-80mps.
>> >
>> > I'm running RabbitMQ 1.6.0. And nether rabbitmq, the consumer, or the
>> > publisher are using more than 40% CPU.
>> >
>> > I assume I shouldn't be seeing this? Any help much appreciated.
>>
>> You didn't include any code, but I'm going to take a stab in the dark
>> anyway. If you are using the ruby amqp gem you might be doing
>> something like this:
>>
>> def next_message
>>   exchange = MQ.fanout 'foo'
>>   queue = MQ.queue 'bar'
>>
>>   queue.bind exchange
>>
>>   newest_message = queue.pop
>> end
>>
>> In the code above the call to MQ.<whatever> is opening a new channel
>> to rabbitmq each time the method is called. You are essentially
>> leaking channels all over the place every time you try to pop a new
>> message.
>>
>> If you aren't using the ruby amqp stuff, I still recommend checking
>> your code for whatever library you chose. You only need to allocate a
>> few channels and reuse them.
>>
>> cr
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>



-- 
http://mixpanel.com
Blog: http://blog.mixpanel.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090907/fe029381/attachment.htm