[rabbitmq-discuss] new queue causes extremely poor disk IO

[Kevin Goess]

Hi, we were hoping to get some pointers in looking into this problem.
 We're running rabbitmq 2.7.1-1 in a xen VM on debian 6 squeeze.

We generally process about 20 messages/sec, with occasional transient
spikes to 500.

We recently added a new queue and saw the disk utilization jump to 60-80%
and the iowait  to 40-60%. Read sectors/sec is around 0, and write
sectors/sec is about 1k, which is unchanged from before adding this new
queue.

The input and output on this new queue is only 1-10 per second.  The
messages in it are only 60-70 bytes.  We currently have a backlog of about
100,000 messages in the queue, we maxed out at 250,000.

The individual consumers of this new queue don't fetch that many
messages--the work they're associated with takes 1-30 seconds to perform.
 We're using STOMP as a message transport.

The problem is that every consumer we add to the queue seems to increase
disk activity by about 10%, and by the time we have 8 consumers, the
rabbitmq server is overloaded enough that other queues start falling behind
in their deliveries.

If we shut off all the consumers on the new queue, disk activity
immediately falls to easily sustainable levels (near zero iowait).

Is there anything we can look at to get further information on the problem?
 Any pointers you have would be helpful.  Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130517/046619ae/attachment.htm>