[rabbitmq-discuss] High I/O On RHEL?

Thu Oct 27 12:25:40 BST 2011

One of my clients has been having a recurring problem of the
filesystems on their brokers randomly becoming readonly. Their admin
investigated the errors and determined it was related to this issue
(http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=51306)
with their running Oracle Enterprise Linux on top of VMWare.

I'm a little confounded and how there could even be too high I/O
interaction for the filesystem to keep up. At most, the brokers only
see 400 messages per second and usually average around 180 messages
per second. This is way below the rates I see on a cluster I run on
Rackspace for another client and I've never seen this happen before.

It's pretty annoying because the filesystem goes readonly randomly in
middle of the night and causes rabbitMQ to shutdown (since having /var
read only screws a lot of things up) and whoever is on call gets paged
(usually at 2am). I've suggested moving the brokers to physical boxes
or using Ubuntu instead of Oracle Enterprise Linux, but I feel like
I'm just making the later suggestion out of spite (I don't like Oracle
Enterprise Linux). :)

Any ideas?

Thanks,
James