[rabbitmq-discuss] millions of unack'd messages in a day -- disk store instead of ram?

[Brian Whitman]

We run rabbitMQ on Amazon EC2 "small" servers (1.7GB ram.) The rabbitMQ
install is out of the box, simply installing from the debian package. We
access rabbit through a python AMQP interface (py-amqplib) and wrapper
client that we set up like this:
http://github.com/bwhitman/py-amqplib-wrapper/tree/master

Our use is probably non-normative: we keep millions of messages around most
of the day. An average message stays in a queue for up to 12 hours, and we
have a dozen or so queues that each reach about 1 million messages at peak.

We quickly determined that a single EC2 small node can support up to about
1-2m un-ack'd messages before it runs out of ram. And when it runs out of
ram the rabbit instance stops responding. To get around this we boot more
rabbit nodes and we shard via a consistent hash that routes messages based
on their ID. This gets us more message capacity but gets expensive and has
more moving parts that can break.

I am wondering if there is a better way. I would gladly take a message
consume speed hit to be able to have the rabbit store backed by disk instead
of keeping it live in ram as it seems to. Is this possible? How do people
set up simple message queues that can store millions of messages for long
periods of time?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090427/51f06e98/attachment.htm