[rabbitmq-discuss] RabbitMQ memory management

[Ben Hood]

Eran,

On Thu, Sep 11, 2008 at 1:53 PM, Eran Sandler <eran.sandler at gmail.com> wrote:
> I prefer not to lose these messages, hence the durable queue, but I can
> recover from not getting them.

OK, then you have a choice :-) You can use Rabbit in resource hungry
way, which is fine or you can manage by exception and recover from
where ever the data came from. The choice is an architecture decision,
so there is no right or wrong way.

> The behavior I would expect, such as flowing back to the disk is to flush
> them to the disk and only keep a small portion in-memory, when "the cows
> come home" (or more exactly, the cowboy comes back and pushes the cows to
> their rightful place) it will start reading from the disk chunk by chunk
> until it reaches a point where all the backlog finished and from there it
> will go back to work as it was before.
>
> Having that combined with a key that says when we should start throwing
> things directly to the disk without keeping them in memory (or without
> keeping them in memory too much) as well as a max queue length (depth,
> depends on where you look at it) would be really great.

Sounds reasonable, so let's hope your disk doesn't fill up :-)

> Having the ability to write a queue plugin would be really great. There is
> always that 20% that requires a certain behavior that is not off the shelf

Sure, and if we had pluggable queues, this kind of requirement could
be tackled without having to hack on the server.

Code and/or design suggestions are welcome :-)

Ben