[rabbitmq-discuss] RabbitMQ memory management

[Ben Hood]

Edwin,

On Fri, Sep 12, 2008 at 2:31 AM, Edwin Fine
<rabbitmq-discuss_efine at usa.net> wrote:
> I think you misunderstood my message. I was saying RabbitMQ should be able
> to store a non-memory-limited number of persistent messages in the absence
> of a consumer draining the queue. So to be exact, these would be
> non-transactional persistent messages. You replied to me about
> non-persistent messages, which I know are memory-limited. I am confused,
> because I was at one time (when I was doing my due diligence) convinced that
> RabbitMQ had no practical limit (other than any set by Mnesia and hard disk
> space) for storing non-transactional persistent messages. I am simply
> concerned that perhaps I was mistaken and was asking you to elaborate.

Thanks for the clarification.

The status quo is that irrespective of whether we are talking about
persistent or non-persistent messages, messages are queued up in
memory until they are drained. The difference between persistent and
non-persistent is that with persistent messages a replica of each
message is written to disk. This disk copy will only be used in a
recovery situation for messages that have not been acknowledged.

Obviously the limit to which you carry on queuing messages without
draining them depends on the physical resources available to the
Erlang VM and how virtual memory is utilized, which in turn depends on
your OS and production setup.

So to draw a line in the sand, you could go through and calibrate your
setup wrt to the point at which it cannot allocate any more memory. By
doing so, you've worked out approximately what your absolute bottom
line will be. If, for example, your expected capacity is a small
fraction of this known limit for your particular setup, and that in
practice, you would notice undrained messages long before you hit the
limit, you *might* decide that for all intents and purposes this is a
acceptable and manageable risk.

Whilst doing your testing, you may not have encountered any problems,
because you may not have pushed it to the limit. All I am saying is
that in doing stress testing for the scalability improvements I am
working on, I have pushed Rabbit to its limits at various stages and
am just informing you of the current theoretical worse case scenario -
ATM queue depth is bounded by memory and there is no overflow
facility.

However, you may be more risk averse and not want to entertain the
possibility of exhausting your system resources, ever.

In that case, as you have rightfully pointed out, an obvious
improvement to Rabbit is to overflow to disk when a certain queue
depth is reached.

This could apply to persistent and non-persistent messages alike.

I'll take this down as an enhancement suggestion which may factored in
the work that we have planned to make queues pluggable. An alternative
is to incorporate this into the current queue implementation, but
whether we do this would basically depend on what resources we have
available and how acute it is time wise.

Another related improvement includes implementing message expiry.

Please let us what your time scales are for when you absolutely need
this in production.

HTH,

Ben