[rabbitmq-discuss] rabbitmq causing heavy disk i/o with nothing in the queues

[Kevin Goess]

On Thu, Jul 11, 2013 at 2:08 PM, Matthias Radestock
<matthias at rabbitmq.com>wrote:

> On 11/07/13 17:38, Kevin Goess wrote:
>
>> Circling back after an earlier question about this problem. We've
>> upgraded to 3.1.3-1.  We've moved the mnesia files to a dedicated 10k
>> sata drive via RABBITMQ_MNESIA_BASE, a new drive that nothing else is
>> using, on a xen virtual machine that is only running rabbitmq.  We
>> generally deal with about 20 messages per second.
>>
>> But we're seeing pretty heavy disk activity, with iowait 30-40%, and
>> iostat numbers like this:
>>
>> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
>> avgrq-sz avgqu-sz   await  svctm  %util
>> xvdbp1            0.00    60.00    0.00  143.40     0.00  1452.80
>>   10.13     0.88    6.14 4.22 60.48
>>
>> Is this expected?  That seems excessive, and I worry that the high
>> iowait could cause us problems as we grow, so I was hoping to get some
>> pointers to things to look at that to improve that.
>>
>
> My suspicious is that somehow your usage pattern results in a lot of
> queue/exchange/binding creation/deletion, which, if the entities involved
> are durable, require disk writes and fsyncs.
>
> To test the validity of that theory, please try to find out what files are
> seeing activity, and report back.
>


This is all the writing that's gone on in the last ten minutes:

$ find . -newer kg-test-timer -type f -ls
262153 10784 -rw-r--r--   1 root     root     11036923 Jul 16 12:31
./msg_store_persistent/41228.rdq
262155 16392 -rw-r--r--   1 root     root     16777655 Jul 16 12:21
./msg_store_persistent/41227.rdq
262274  120 -rw-r--r--   1 rabbitmq rabbitmq   116640 Jul 16 12:30
./queues/8EP2UVB3Z8SAIWFBFKQ0100K4/journal.jif
262166   32 -rw-r--r--   1 root     root        28222 Jul 16 12:28
./queues/9K0C0VGF04K4O788HH6ZFMDAG/4747.idx
262204   24 -rw-r--r--   1 rabbitmq rabbitmq    20000 Jul 16 12:30
./queues/9K0C0VGF04K4O788HH6ZFMDAG/journal.jif
262228    0 -rw-r--r--   1 rabbitmq rabbitmq        0 Jul 16 12:27
./queues/A6K4C7JLF1P0AHEXXN9633747/journal.jif
262172    4 -rw-r--r--   1 root     root           28 Jul 16 12:26
./queues/C8NMZYECUT2TCZUNODNUQ122Y/0.idx
262224    0 -rw-r--r--   1 rabbitmq rabbitmq        0 Jul 16 12:26
./queues/C8NMZYECUT2TCZUNODNUQ122Y/journal.jif
262291  136 -rw-r--r--   1 rabbitmq rabbitmq   134008 Jul 16 12:31
./queues/5W6SPLYHNTA4ZH2CONMRXOT90/journal.jif
262161   12 -rw-r--r--   1 root     root         8256 Jul 16 12:30
./queues/3GMBIL72D6Q6F4LC6TWTYCPWA/1.idx
262236    0 -rw-r--r--   1 rabbitmq rabbitmq        0 Jul 16 12:30
./queues/3GMBIL72D6Q6F4LC6TWTYCPWA/journal.jif
262234  424 -rw-r--r--   1 rabbitmq rabbitmq   430024 Jul 16 12:31
./queues/ERZ5DFKEUXBK1QJQJLBQN3KFI/journal.jif
262199  528 -rw-r--r--   1 rabbitmq rabbitmq   536208 Jul 16 12:30
./queues/DWEIOI0J2S6CMT74JYWUZXVL8/journal.jif
262282   28 -rw-r--r--   1 rabbitmq rabbitmq    21600 Jul 16 12:30
./queues/1XAUC81LT5GM7ZI05RROF4KL0/journal.jif
262297    0 -rw-r--r--   1 rabbitmq rabbitmq        0 Jul 16 12:19
./queues/31J4BX7549NFNZ1Q6XH1Z19MJ/journal.jif


and that list is almost identical to that for the last *hour*, with the
addition of these files

262152   80 -rw-r--r--   1 rabbitmq rabbitmq    78942 Jul 16 11:53
./msg_store_persistent/17778.rdq
262321    0 -rw-r--r--   1 rabbitmq rabbitmq        0 Jul 16 11:36
./queues/AVS8NDWHNCZ8X0Y2599HSOVJ0/journal.jif
262341    0 -rw-r--r--   1 rabbitmq rabbitmq        0 Jul 16 11:42
./queues/U7FN5251IRUUH23XQIPFTBND/journal.jif


Does that mean anything to you?  Is there a way we can match those
directory names with queues?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130716/5b69d6aa/attachment.htm>