[rabbitmq-discuss] Persister crashes Rabbit

[Ilya Grigorik]

I'm doing some load testing on Rabbit and I'm bumping into the following
scenario:

Fanout exchange > Durable queue

I'm pushing ~300~350 message/s into Rabbit, each ~10kb/s. Watching the
network pipe, the input is fluctuating around 3.5mb/s. Comparing these
numbers to the docs, they seem to be consistent with the documented number
of 4000m/s at 1kb each. (or at least in the same ballpark).

Rabbit seems to become unstable once the persister.LOG crosses the 1GB
limit. Disk syncs are taking longer and longer (for obvious reasons), and
memory starts to fluctuate all over the map - from 2GB to 7GB (I am seeing
high memory watermark warnings). Then, without any warnings in the log file,
the process falls over and everyone gets disconnected. On the rabbit node, I
still see epmd and beam.smp processes running, but the node is reported as
down. Running the node in the foreground, I get:

broker running
/usr/local/lib/erlang/lib/os_mon-2.1.8/priv/bin/memsup: Erlang has closed.
Erlang has closed

(Tried disabling memsup and lowering the mem watermark to 0.7, same thing..
Erlang closed)

Looking at the persister logs (no .previous):
-rw-r--r-- 1 root root 2.0G Nov 15 10:24 rabbit_persister.LOG

At this point I can't even restart the node. The only way to get it back up
is to remove the persister logs and let is start clean.

I've scanned the irc logs [1], and it does seem that persister is still up
in the air.. Having said that, any ideas or recommendations?

[1] http://dev.rabbitmq.com/irclog/index.php?date=2008-07-31

P.S. Running latest version of RabbitMQ from trunk (as of Nov 14th), erl:
R12B-5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20081115/25b58c25/attachment.htm