[rabbitmq-discuss] Persister crashes Rabbit

[Ilya Grigorik]

>
> What client are you using to publish the messages?
>

I'm using Aman's ruby library [1], and a simple publisher client. [2]

[1] http://github.com/tmm1/amqp
[2] http://pastie.org/315607

Rabbit seems to become unstable once the persister.LOG crosses the 1GB
> limit. Disk syncs are taking longer and longer (for obvious reasons), and
> memory starts to fluctuate all over the map - from 2GB to 7GB (I am seeing
> high memory watermark warnings). Then, without any warnings in the log file,
> the process falls over and everyone gets disconnected.
>
> Persistent messages still have copies in memory, so if you have no
> consumers or the rate of consumption is lower than the publishing rate
> eventually rabbit will run out of memory.
>

So the messages don't get flushed out to disk at all? Right now, I'm running
two publishers and no consumers.. I would expect RabbitMQ to keep a stack of
messages in memory and overflow to disk - is this wrong? Hmm, now that I
think about it, this would explain the log rollover pattern I'm seeing:
.previous is removed from disk, and full log file is dumped from memory. I
was surprised by this pattern, but if it's always doing a full memory flush,
that would make sense.

- That would also mean that the number of messages we can queue is
essentially limited by the amount of available RAM?
- Is there any way to do an incremental append to the logfile, instead of a
full flush?
- I've seen a few mentions of RDBMS hooks for persistence, how far away are
we from this?

P.S. Running latest version of RabbitMQ from trunk (as of Nov 14th),
>
> The flow control in that version should tell the producers to stop sending
> messages when memory becomes critical. However, it does require a
> cooperating client. Hence my question above about what client you are using.
> At present the trunk version of the Java client is the only one that handles
> flow control correctly.
>

Ah, interesting, I'll ping Aman on that.

Cheers,
Ilya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20081115/312a1d45/attachment.htm