[rabbitmq-discuss] Roadmap Questions

Wed Jun 3 23:24:32 BST 2009

Keith,

Keith Irwin wrote:
> For instance, here's what I'd like to see:
> 
>  * Can store a LOT of messages on disk (more than fits in memory).
> 
>  * When one of the nodes in a cluster goes down, the queues on that 
> cluster begin working on another node. In other words, a fatal crash of 
> a node (server goes up in flames, disk melts) does not result in message 
> loss. (Replicated queues?)

The two key features of the new persister are

- message storage space is bounded by disk space rather than available 
physical or virtual memory

- the computational cost of keeping a message stored is bounded

So these address your first point.

Your second point is only tangentially related to persistence - one can 
easily image a system where message loss is prevented by keeping a 
sufficient number of copies of the message in memory on different 
machines, without the messages ever hitting the disk.

There are actually some fairly easy ways to get the kind of reliability 
you refer to from the existing RabbitMQ broker. Just to mention two of 
the options: 1) get the persister to write to a SAN, such that in the 
event of a node failure a standby node can pick up from where the first 
node left off, 2) route the messages to multiple queues on different 
nodes and get the consumers to consume from all the queues and perform 
deduplication.

We have discussed these and other options with users on a number of 
occasions. Alexis may have some slides handy that illustrate the 
different approaches.

Regards,

Matthias.