I'm doing some load testing on Rabbit and I'm bumping into the following scenario:<br>
<br>
Fanout exchange > Durable queue<br>
<br>
I'm pushing ~300~350 message/s into Rabbit, each ~10kb/s. Watching the network pipe, the input is fluctuating around 3.5mb/s. Comparing these numbers to the docs, they seem to be consistent with the documented number of 4000m/s at 1kb each. (or at least in the same ballpark).<br>
<br>
Rabbit seems to become unstable once the persister.LOG crosses the 1GB limit. Disk syncs are taking longer and longer (for obvious reasons), and memory starts to fluctuate all over the map - from 2GB to 7GB (I am seeing high memory watermark warnings). Then, without any warnings in the log file, the process falls over and everyone gets disconnected. On the rabbit node, I still see epmd and beam.smp processes running, but the node is reported as down. Running the node in the foreground, I get:<br>
<br>
broker running<br>
/usr/local/lib/erlang/lib/os_mon-2.1.8/priv/bin/memsup: Erlang has closed.<br><div id=":kv" class="ArwC7c ckChnd">
Erlang has closed<br>
<br>
(Tried disabling memsup and lowering the mem watermark to 0.7, same thing.. Erlang closed)<br>
<br>
Looking at the persister logs (no .previous):<br>
-rw-r--r-- 1 root root 2.0G Nov 15 10:24 rabbit_persister.LOG<br>
<br>
At this point I can't even restart the node. The only way to get it back up is to remove the persister logs and let is start clean.<br>
<br>
I've scanned the irc logs [1], and it does seem that persister is still up in the air.. Having said that, any ideas or recommendations?<br>
<br>
[1] <a href="http://dev.rabbitmq.com/irclog/index.php?date=2008-07-31" target="_blank">http://dev.rabbitmq.com/irclog/index.php?date=2008-07-31</a><br>
<br>
P.S. Running latest version of RabbitMQ from trunk (as of Nov 14th), erl: R12B-5</div>