[rabbitmq-discuss] Erlang client: function_clause error

[Ben Hood]

Valentino,

On Mon, Oct 27, 2008 at 11:01 PM, Valentino Volonghi <dialtone at gmail.com> wrote:
> Speaking of which... Is there any way (donations, manual labor, slavery or
> such) that
> I can be of help to change the queue state to a new object that knows how to
> persist
> without using all the memory up?

In general, you can help by either contributing code yourself or by
financing the reprioritization of the roadmap.

Disk overflow or queue paging is on the mid term roadmap as something
we are going to do, but we still need to gather requirements.

Here are a few examples:

- When do you decide to page things to disk?
- Is it done on memory consumption or queue depth?
- Is this configurable per queue or across the broker?
- What are sensible defaults so that people who haven't even though
  about paging don't get affected by overly-agressive defaults?
- When and how do you swap back in - is this automatic or manual?
- How do you decide when the low water mark has been reached after
  having commenced the page-in? Do you resume flow control at this point?
- If you do page, are you interested in last image caching?
- Do you want to apply application level heuristics to selectively purge
  overflowed queues?
- What role do TTLs play in this scenario?
- Furthermore, we do already have a fast message persister - it's just
  that it's geared to write as quickly as possible, not read.
- Ask yourself, if we implement paging, is are we potentially reinventing
  a wheel that the OS has already invented?
- And as always, what do you do when your SAN fills up?

One suggestion to kick things off is to begin a more structured
analysis of the whole problem on the wiki and start a dedicated
discussion thread around this. For example, one could start a document
highlighting the motivation and requirements and let interested
parties comment on this.

> From what I see by reading the code I think that the change would be
> isolated inside
> rabbit_amqqueue_process.erl and the exact variable is message_buffer. An
> object
> with a similar API but a different storage strategy could maybe be swapped
> in instead
> of the current queue.

True - the intention is to have a code base that is as short as
possible so that it can be easily understood and adapted.

Having said that, whilst it may be straight forward to do a hello
world overflow, the devil is in the detail of all of the moving parts
and different scenarios that you have to account for whilst maintaing
the clarity of the code base. And testing it of course.

HTH,

Ben

Ben