[rabbitmq-discuss] High I/O On RHEL?

Tue Nov 1 12:40:54 GMT 2011

On Thu, Oct 27, 2011 at 12:25, James Carr <james.r.carr at gmail.com> wrote:
> One of my clients has been having a recurring problem of the
> filesystems on their brokers randomly becoming readonly. Their admin
> investigated the errors and determined it was related to this issue
> (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=51306)
> with their running Oracle Enterprise Linux on top of VMWare.
>
> I'm a little confounded and how there could even be too high I/O
> interaction for the filesystem to keep up. At most, the brokers only
> see 400 messages per second and usually average around 180 messages
> per second. This is way below the rates I see on a cluster I run on
> Rackspace for another client and I've never seen this happen before.
>
> It's pretty annoying because the filesystem goes readonly randomly in
> middle of the night and causes rabbitMQ to shutdown (since having /var
> read only screws a lot of things up) and whoever is on call gets paged
> (usually at 2am). I've suggested moving the brokers to physical boxes
> or using Ubuntu instead of Oracle Enterprise Linux, but I feel like
> I'm just making the later suggestion out of spite (I don't like Oracle
> Enterprise Linux). :)
>
> Any ideas?

I have no clue how to solve the problem. But I have some
ideas about possible workarounds:
 a) maybe use ext2? (basically - a fs without journaling)
 b) take a look at `errors=continue` option in `man mount`

Cheers,
  Marek