[rabbitmq-discuss] LATEST.LOG growth on clustered ram node

Tue Jun 21 16:18:36 BST 2011

> On Mon, Jun 20, 2011 at 06:24:45PM -0400, Chris Madden wrote:
> > In a nutshell, the LATEST.LOG file on my ram clustered node grows, by
> > all accounts, forever. I left it running over the weekend and it was
> > well over 500meg. In the past I've let it run until it consumed all
> > the space on the partition and the broker instance cratered. Running
> > it as a ram node instead of a disc node obviates the problem, but is
> > sub-optimal.
> > 
> > I'm using rabbit 2.5.0, though I've seen the problem on 2.4.0 and
> > 2.4.1. The OS is linux based (2.6.32), and our Erlang version is R14A.
> 
> Very interesting. I've not heard of that one before and it certainly
> points to an mnesia issue. R14A is a beta release and I would strongly
> point an accusing finger at that first. Please try the latest Erlang
> R14B03.
> 
> Matthew
> 
> 
 I upgraded our erlang version to R14B03, but the LATEST.LOG still continues to be growing, albeit, more slowly perhaps? 

I'm kind of suspicious of one of the queue usage models I'm using. When one of the processes needs to make a one-off request (rpc like), it creates a queue dedicated to that request, and expects the reply to come back on that queue. After a timeout or the response is delivered, the queue is taken down. As a result, we have queues come and go a few times a minute... not really ideal, but it helps with debugging. Beyond this, I think the rabbit setup is pretty standard. Any ideas?

Below is the status/cluster_status output, in case it points at something obvious:

drone:0:~ # rabbitmqctl status
Status of node rabbit at drone ...
[{pid,12348},
{running_applications,[{rabbit,"RabbitMQ","2.5.0"},
{os_mon,"CPO CXC 138 46","2.2.6"},
{sasl,"SASL CXC 138 11","2.1.9.4"},
{mnesia,"MNESIA CXC 138 12","4.4.19"},
{stdlib,"ERTS CXC 138 10","1.17.4"},
{kernel,"ERTS CXC 138 10","2.14.4"}]},
{os,{unix,linux}},
{erlang_version,"Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,[{total,30529744},
{processes,10871688},
{processes_used,10849544},
{system,19658056},
{atom,1114745},
{atom_used,1105101},
{binary,4144112},
{code,11021554},
{ets,1404144}]}]
...done.

drone:0:~ # rabbitmqctl cluster_status
Cluster status of node rabbit at drone ...
[{nodes,[{disc,[rabbit at controller]},{ram,[rabbit at drone]}]},
{running_nodes,[rabbit at controller,rabbit at drone]}]
...done.