[rabbitmq-discuss] Is there a channel publish throughput limit to a queue?

Wed Oct 5 10:13:21 BST 2011

On 04/10/11 19:00, Allen Kim wrote:
> Thanks for your reply. I have been waiting for someone to reply.
>
> 1. They are persistent messages (disk writing speed could slow down that
> much?)

A good disk can sync at about 200-300 syncs per second. If each 
published message is being synced to disk you should expect that level 
of performance. That's a hardware limit, not RabbitMQ related.

If you want to go above that and still have your data actually persisted 
when you think it is (i.e. when the confirm is returned), then you 
either need to get a solid state disk, SAN, or disks with non volatile 
cache.

Most desktop setups have disk write cache enabled by default, so you can 
get falsely highly rates of several thousand "syncs" per second when 
running on your cheap desktop/laptop only to be surprised when the rate 
goes down when you deploy to your production servers (since prod servers 
generally don't have cache enabled by default).

> 2. NOT large message (it's around two lines 200-400 bytes)
> 3. routing to only one queue
> 4. NO transaction
> 5. 3-4 consumers with basic.qos prefetch to 100.
>
> So, #1 and #5 matters?
>
> Allen
>
>
>
> On 11-10-04 4:37 AM, "Matthias Radestock"<matthias at rabbitmq.com>  wrote:
>
>> On 29/09/11 21:47, Allen Kim wrote:
>>> I checked the channel, and found out that the channel was still
>>> publishing to a queue.
>> Looks like the queues couldn't keep up with the inbound publishing rate,
>> so rabbit ended up buffering messages in the channel.
>>
>>> The speed of publishing was around 700/s to 800/s, which is definitely
>>> not 2000/s.
>> ...which is another indication of the above.
>>
>>> I was wondering if there is limit that affects publishing speed.
>> Well, obviously rabbit cannot process messages at an infinite rate, so
>> clearly there has to be a limit ;)
>>
>> However, 700Hz sounds quite low. On my desktop I can get more than 20
>> times that. But there are many factors which can reduce that rate:
>> - persistent messages
>> - large messages
>> - routing messages to lots of queues
>> - using transactions
>> - having lots of consumers with basic.qos prefetch
>> etc.
>>
>>
>> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss