[rabbitmq-discuss] rabbitmq causing heavy disk i/o with nothing in the queues

[Kevin Goess]

Circling back after an earlier question about this problem. We've upgraded
to 3.1.3-1.  We've moved the mnesia files to a dedicated 10k sata drive
via RABBITMQ_MNESIA_BASE, a new drive that nothing else is using, on a xen
virtual machine that is only running rabbitmq.  We generally deal with
about 20 messages per second.

But we're seeing pretty heavy disk activity, with iowait 30-40%, and iostat
numbers like this:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
xvdbp1            0.00    60.00    0.00  143.40     0.00  1452.80    10.13
    0.88    6.14   4.22  60.48

Is this expected?  That seems excessive, and I worry that the high iowait
could cause us problems as we grow, so I was hoping to get some pointers to
things to look at that to improve that.  Any information, tips, or
suggestions would be much appreciated!

There's nothing in the sasl logs, and the rabbitmq logs are nothing but
these:

INFO REPORT==== 11-Jul-2013::09:25:55 ===
accepting STOMP connection <0.14174.238> (...)

=INFO REPORT==== 11-Jul-2013::09:26:12 ===
closing STOMP connection <0.13749.238> (...)

These are the queues we have in there now:

$ rabbitmqctl list_queues name durable arguments pid messages_ready
Listing queues ...
article.added_to_content_bundle true [] <rabbit at rabbitmq.1.239.0> 0
institution.added_to_content_bundle true [] <rabbit at rabbitmq.1.215.0> 0
production.article_downloads true [] <rabbit at rabbitmq.1.221.0> 0
production.article_downloads.error true [] <rabbit at rabbitmq.1.265.0> 0
production.article_downloads.unaggregated true [] <rabbit at rabbitmq.1.274.0>
0
production.article_downloads.unaggregated.error true []
<rabbit at rabbitmq.1.294.0> 0
production.bulk_email true [] <rabbit at rabbitmq.1.273.0> 0
production.bulk_email.error true [] <rabbit at rabbitmq.1.287.0> 0
production.certify-existing-expedites true [] <rabbit at rabbitmq.1.307.0> 0
production.certify-existing-expedites.error true [] <rabbit at rabbitmq.1.253.0
> 0
production.create-sw-site true [] <rabbit at rabbitmq.1.291.0> 0
production.create-sw-site.error true [] <rabbit at rabbitmq.1.282.0> 0
production.dcnetwork_follow true [] <rabbit at rabbitmq.1.300.0> 0
production.dcnetwork_follow-error true [] <rabbit at rabbitmq.1.289.0> 229
production.dcnetwork_follow.error true [] <rabbit at rabbitmq.1.232.0> 0
production.dcnetwork_update true [] <rabbit at rabbitmq.1.219.0> 0
production.dcnetwork_update.error true [] <rabbit at rabbitmq.1.301.0> 0
production.email true [] <rabbit at rabbitmq.1.223.0> 0
production.email.error true [] <rabbit at rabbitmq.1.241.0> 1
production.expresso-archive-pending-articles true []
<rabbit at rabbitmq.1.305.0> 0
production.expresso-archive-pending-articles.error true []
<rabbit at rabbitmq.1.299.0> 0
production.expresso-copy-article true [] <rabbit at rabbitmq.1.244.0> 0
production.expresso-copy-article.error true [] <rabbit at rabbitmq.1.242.0> 1
production.general-errors true [] <rabbit at rabbitmq.1.250.0> 150
production.generate-usage-report true [] <rabbit at rabbitmq.1.235.0> 0
production.generate-usage-report.error true [] <rabbit at rabbitmq.1.222.0> 0
production.guest_access_report true [] <rabbit at rabbitmq.1.238.0> 0
production.guest_access_report.error true [] <rabbit at rabbitmq.1.281.0> 0
production.logged_request true [] <rabbit at rabbitmq.1.302.0> 0
production.logged_request.error true [] <rabbit at rabbitmq.1.245.0> 0
production.migrate-article-editor-notes true [] <rabbit at rabbitmq.1.14254.0>
0
production.migrate-article-editor-notes.error true []
<rabbit at rabbitmq.1.14255.0> 96
production.mq-solr-update-article true [] <rabbit at rabbitmq.1.256.0> 0
production.mq-solr-update-article.error true [] <rabbit at rabbitmq.1.275.0>
185
production.mq-solr-update-article.low-priority true []
<rabbit at rabbitmq.1.262.0> 0
production.mq-solr-update-article.low-priority.error true []
<rabbit at rabbitmq.1.259.0> 3
production.publish-context true [] <rabbit at rabbitmq.1.295.0> 0
production.publish-context.error true [] <rabbit at rabbitmq.1.268.0> 0
production.purge-stamped-files true [] <rabbit at rabbitmq.1.14256.0> 0
production.purge-stamped-files.error true [] <rabbit at rabbitmq.1.14257.0> 12
production.record_download true [] <rabbit at rabbitmq.1.257.0> 0
production.record_download.error true [] <rabbit at rabbitmq.1.285.0> 0
production.saved_search_results true [] <rabbit at rabbitmq.1.286.0> 0
production.saved_search_results.error true [] <rabbit at rabbitmq.1.230.0> 0
production.solr-update-article true [] <rabbit at rabbitmq.1.217.0> 0
production.stamp-pdf true [] <rabbit at rabbitmq.1.225.0> 0
production.stamp-pdf.error true [] <rabbit at rabbitmq.1.251.0> 398
production.stamp-pdf.low-priority true [] <rabbit at rabbitmq.1.276.0> 0
production.stamp-pdf.low-priority.error true [] <rabbit at rabbitmq.1.236.0> 0
production.test true [] <rabbit at rabbitmq.1.233.0> 0
production.test.error true [] <rabbit at rabbitmq.1.254.0> 0
production.withdraw-for-author true [] <rabbit at rabbitmq.1.296.0> 0
production.withdraw-for-author.error true [] <rabbit at rabbitmq.1.308.0> 0
test true [] <rabbit at rabbitmq.1.279.0> 0
user.added_to_content_bundle true [] <rabbit at rabbitmq.1.237.0> 0
...done.


-- 
Kevin M. Goess
Software Engineer
Berkeley Electronic Press
kgoess at bepress.com

510-665-1200 x179
www.bepress.com

bepress: sustainable scholarly publishing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130711/a668519b/attachment.htm>