[rabbitmq-discuss] RabbitMQ throughput

[Suhail Doshi]

Turns out I already output info to basically notice if the consumer is
blocking:
2009-10-01 20:55:27,935 INFO [id: 99704402] Processing complete,
acknowledged.
2009-10-01 20:55:27,939 INFO [id: 99704402] Received queue item,
processing...

2009-10-01 20:55:29,215 INFO [id: 99704402] Processing complete,
acknowledged.
2009-10-01 20:55:29,215 INFO [id: 99704402] Received queue item,
processing...

You'll notice there's a gap between receiving and processing so that means
the consumer is just waiting for an item. So it really does look like
rabbitmq is not able to get items to me fast enough with lots of items
backed in the queue.

Suhail

On Thu, Oct 1, 2009 at 1:40 PM, Suhail Doshi <suhail at mixpanel.com> wrote:

>
>
> On Thu, Oct 1, 2009 at 1:32 PM, Matthias Radestock <matthias at lshift.net>wrote:
>
>> Suhail,
>>
>> Suhail Doshi wrote:
>>
>>> Does anyone know what rabbitmq throughput is from items sitting in the
>>> queue to sending them to a consumer?
>>>
>>> Lets say we're doing 200 items in the queue a second with 50 consumers
>>> waiting to process the item, usually they do in < 30ms.
>>>
>>
>> So that's ~7kHz, which doesn't look unreasonable, but there are many
>> factors at play here which we'd need to know about in order to make a better
>> assessment:
>>
>> - are these messages persistent or transient? Persistent
>
> - how big are the messages? Less than 500 bytes usually
>> - do the consumer consume in no-ack mode, or ack mode? ack mode
>> - are the consumers sharing a connection (or even channel) or do they all
>> have their separate connection/channel? do not share a channel/connection
>> - where are the consumers in the network topology in relation to the
>> server? very close, same rack presumably
>> - what's the hardware spec of the server machine? quad core, 4 GB of RAM
>> - what version of Erlang are you running? 5.7.2
>> - what AMQP client are you using? Python library created by Barry Pederson
>>
>> Also, ...
>>
>>  Watching our consumer logs, it feels like they don't get items fast
>>> enough from rabbitmq, is that possible or is it me?
>>>
>>
>> Can you instrument the consumers to measure how long they are blocked
>> waiting for messages from RabbitMQ? Yes possibly, I should check you're
>> right.
>>
>> Also, with 200 messages shared across 50 consumers each consumer will only
>> get four items, which is a rather small figure to do any meaningful
>> statistical analysis with.
>>
>>
>> Regards,
>>
>> Matthias.
>>
>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>



-- 
http://mixpanel.com
Blog: http://blog.mixpanel.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20091001/dccafcca/attachment.htm