[rabbitmq-discuss] RDQ files filling the disk.

[Tim Watson]

On 21 May 2013, at 23:19, "michi.oshima" <moshima at advent.com> wrote:

> We recently had an issue where Rabbit1 started taking up a lot of disk
> space.  A directory called 'msg_store_transient' got up to some 20 GB,
> filled with files with *.rdq extension.  They are about 16MB each in size. 
> It almost filled up the disk (it's a small disk) and caused our application
> (client to Rabbit1) to fail.
> 

Rabbit will start paging transient (non-persistent) messages to disk if memory starts to get tight (i.e., close to the high watermark) and they'll end up in that file.

> Meanwhile, Rabbit2 reportedly have been receiving the messages from Rabbit1. 
> I also saw no shovel connection related errors in rabbit.log, etc.
> 
> Q1. Does this sound plausible?  If so, how could this happen?

Sounds like rabbit1 is paging due to memory pressure. Do you see a memory alarm in the logs? Even if you don't, it may be paging pre-emptively.

> Q2. I'm thinking the rdq files are meant to get deleted at some point.  What
> would cause the files to *not* get deleted?

The paged out messages will need to be delivered and acked and, at the same time, memory use will need to drop back so that eventually the broker stops paging deliberately. There are probably more specific indicators (I'm not looking at the source right now), but  first of all let's take a look at memory use on that box whilst the file is growing.

Cheers,
Tim