[rabbitmq-discuss] RabbitMQ memory management

[Alexis Richardson]

Hi all,

I have been following this discussion on the list.  To recap:

1 - the overhead of storing a message in rabbitmq is very low meaning
that the case 'broker has filled up, I need more nodes' is quite
unusual and certainly manageable
2 - anyone who needs more queue memory can add more nodes to the broker
cluster transparently
3 - durable/persistent messages are also stored on disk for recovery purposes

If you have slower consumers than producers, then either:
a) #queues will grow in size
or:
b) at least one queue will grow in size.

Note that case (a) is solved by 2 above.  Add more nodes.  How often would you
have to add more nodes?  Due to 1, you can work this out based on your message
size.  For almost all use cases the consumers will have to lag
producers by several
days.  Think about it.  And don't forget you can add more consumers.

Let's say that you cannot add more nodes and you cannot add more consumers.
Assume also that rabbitmq is set up to persist messages to disk for recovery per
point 3 above, but that due to your application you need copies of each message
in memory.  Finally assume you have only one queue, many fast producers, and
one slow consumer.  One example would be SMS.  Even a small old server could
store millions of SMS messages without any of them being consumed.  So that
is your scenario.

[ the above is extreme but the problem use cases are variants of this ]

In cases like this we would need to do one of the following:

* provide a means to tell producers to back off, or alert an operator
* or, implement a system that created more queues and nodes for you,
if one queue became ginormous
* implement either rolling buffers, or agents, that flushed/paged
messages to pluggable stores or directly to disk
* .. or (more extremely) implement rolling buffers that deleted old
unconsumed messages completely when queue size maxed out, and notified
consumers and operators as needed
* some other more fancy flow control TBD
* OR - recommend the application deigner thinks about why their queues
are growing to very large size and if this can be prevented through
application changes

You'll appreciate that the first couple of fixes in this list are
fairly simple to implement, with potential difficulties increasing as
you go down the list ;-)

Yet despite all this, in summary, it is reasonable for users to ask
for a solution that is more 'self healing'.

So - any takers?  Comments, thoughts?

alexis


On Sat, Sep 13, 2008 at 5:00 AM, Matthias Radestock <matthias at lshift.net> wrote:
> Hi,
>
> tsuraan wrote:
>> If RabbitMQ crashes because it's out of memory, I understand that it
>> should be able to start again without losing any data
>
> Only *durable* queues and exchanges, and *persistent* messages are
> recoverable, as per the spec.
>
>> Will the next message sent to it (before any messages are dequeued)
>> cause the queue to crash again? I assume that must be the case, since
>> nothing was lost when the program crashed.
>
> Messages aren't the only entities consuming memory. I would hope that
> RabbitMQ is able to recover the state (as defined above) after an OoM
> error, and take in a few more messages, but it's not something we have
> tested. It's possible that the startup sequence uses just that little
> bit more memory which would push the system over the edge.
>
> Generally, if there are worries about RabbitMQ running out of memory one
> should try catching that case well before it happens. It's not really
> advisable to run a system so close to the limits.
>
>
> Matthias.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>