[rabbitmq-discuss] RabbitMQ memory management

[Ben Hood]

Alexis,

On Sat, Sep 13, 2008 at 10:07 AM, Alexis Richardson
<alexis.richardson at cohesiveft.com> wrote:
> Note that case (a) is solved by 2 above.  Add more nodes.  How often would you
> have to add more nodes?  Due to 1, you can work this out based on your message
> size.  For almost all use cases the consumers will have to lag
> producers by several
> days.  Think about it.  And don't forget you can add more consumers.

Good point.

The main reason why I asked Edwin about his realsitic expectations
surrounding volumetrics was to find out what the breaking point was
for a simple OTS Rabbit installation to do a *very* naive reality
check.

So in the absense of better knowledge, I just thought to myself that
an SMS is roughly 160 bytes long (160 chars with an encoding that is
something less than 8 bit/char plus some routing headers) and just
created an infinite loop to publish them. A sinlge instance of Rabbit
got overfed after publishing 2.5 million of these messages (on a
simple pizzabox setup).

So under the assumption that you may also use more than one logical
queue (by way of natural application partitioning), you may be
spreading the total system load over multiple queues that reside in
memory on different nodes.

In the degenerate case that you send 1 million messages per day to a
single instance, you still have a day and a bit to find some way to
drain the queues. Presumeably, if no SMS's were getting delivered to
the downstream consumers over the course of a day, somebody would
start to care about the fact that the system wasn't actually doing
something. This person would still have a fair amount of time to find
out what is going on and drain the queues before resource consumption
becomes acute.

Ben