<div>We run rabbitMQ on Amazon EC2 "small" servers (1.7GB ram.) The rabbitMQ install is out of the box, simply installing from the debian package. We access rabbit through a python AMQP interface (py-amqplib) and wrapper client that we set up like this: <a href="http://github.com/bwhitman/py-amqplib-wrapper/tree/master">http://github.com/bwhitman/py-amqplib-wrapper/tree/master</a></div>
<div><br></div><div>Our use is probably non-normative: we keep millions of messages around most of the day. An average message stays in a queue for up to 12 hours, and we have a dozen or so queues that each reach about 1 million messages at peak.</div>
<div><br></div><div>We quickly determined that a single EC2 small node can support up to about 1-2m un-ack'd messages before it runs out of ram. And when it runs out of ram the rabbit instance stops responding. To get around this we boot more rabbit nodes and we shard via a consistent hash that routes messages based on their ID. This gets us more message capacity but gets expensive and has more moving parts that can break.</div>
<div><br></div><div>I am wondering if there is a better way. I would gladly take a message consume speed hit to be able to have the rabbit store backed by disk instead of keeping it live in ram as it seems to. Is this possible? How do people set up simple message queues that can store millions of messages for long periods of time?</div>
<div><br></div><div><br></div>
<div><br></div><div><br></div>