[rabbitmq-discuss] millions of unack'd messages in a day -- disk store instead of ram?

[Alexis Richardson]

Brian

Your question is timely.  We are working on a feature in which
persistent messages are stored only on disk, instead of concurrently
on disk and in ram.  If you peruse the archives you will see some info
about this (eg timelines).

You may find this blog post relevant:
http://www.lshift.net/blog/2009/04/02/cranial-surgery-giving-rabbit-more-memory

What disk store do you use on Amazon?  Is it S3 or EBS?

alexis



On Mon, Apr 27, 2009 at 3:00 PM, Brian Whitman <brian at echonest.com> wrote:
> We run rabbitMQ on Amazon EC2 "small" servers (1.7GB ram.) The rabbitMQ
> install is out of the box, simply installing from the debian package. We
> access rabbit through a python AMQP interface (py-amqplib) and wrapper
> client that we set up like
> this: http://github.com/bwhitman/py-amqplib-wrapper/tree/master
> Our use is probably non-normative: we keep millions of messages around most
> of the day. An average message stays in a queue for up to 12 hours, and we
> have a dozen or so queues that each reach about 1 million messages at peak.
> We quickly determined that a single EC2 small node can support up to about
> 1-2m un-ack'd messages before it runs out of ram. And when it runs out of
> ram the rabbit instance stops responding. To get around this we boot more
> rabbit nodes and we shard via a consistent hash that routes messages based
> on their ID. This gets us more message capacity but gets expensive and has
> more moving parts that can break.
> I am wondering if there is a better way. I would gladly take a message
> consume speed hit to be able to have the rabbit store backed by disk instead
> of keeping it live in ram as it seems to. Is this possible? How do people
> set up simple message queues that can store millions of messages for long
> periods of time?
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>