[rabbitmq-discuss] new queue causes extremely poor disk IO

[Tim Watson]

Hi Kevin,

There have been innumerable bug fixes and performance improvements to rabbit since 2.7.1 - is upgrading a possibility?

Cheers,
Tim

On 17 May 2013, at 16:38, Kevin Goess wrote:

> Hi, we were hoping to get some pointers in looking into this problem.  We're running rabbitmq 2.7.1-1 in a xen VM on debian 6 squeeze.
> 
> We generally process about 20 messages/sec, with occasional transient spikes to 500.
> 
> We recently added a new queue and saw the disk utilization jump to 60-80% and the iowait  to 40-60%. Read sectors/sec is around 0, and write sectors/sec is about 1k, which is unchanged from before adding this new queue.
> 
> The input and output on this new queue is only 1-10 per second.  The messages in it are only 60-70 bytes.  We currently have a backlog of about 100,000 messages in the queue, we maxed out at 250,000.
> 
> The individual consumers of this new queue don't fetch that many messages--the work they're associated with takes 1-30 seconds to perform.  We're using STOMP as a message transport.
> 
> The problem is that every consumer we add to the queue seems to increase disk activity by about 10%, and by the time we have 8 consumers, the rabbitmq server is overloaded enough that other queues start falling behind in their deliveries.  
> 
> If we shut off all the consumers on the new queue, disk activity immediately falls to easily sustainable levels (near zero iowait).
> 
> Is there anything we can look at to get further information on the problem?  Any pointers you have would be helpful.  Thanks!
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss