[rabbitmq-discuss] EXT :Re: Memory not flushing to disk

[Emile Joubert]

Hi Logan,

On 23/04/13 18:11, Rodrian, Logan P (IS) wrote:

> Let me present our use case in a larger context and see what your
> thoughts are.  Our goal with using RabbitMQ as a broker was to
> provide a managed "disk backup" of data while consumers were
> unavailable, ensuring no data is lost.  In our environment, the
> consumers may not be available for a long time.

Can you place an upper limit on "long"?

> I understand the use of vm_memory_high_watermark to put a "band-aid"
> on the problem, but it sounds like we will still run into issues
> eventually.
> 
> So, are there any parameters that can be set on the server to
> configure it for this type of environment?  This includes use of any
> alternative plugins that keep the indexes in a different way and/or
> persist data to the disk using a different approach.

There is at least one alternative message index storage plugin:
 http://hg.rabbitmq.com/rabbitmq-toke/

that uses a driver for Tokyo Cabinet:
 http://hg.rabbitmq.com/toke/

This plugin has not reached official status:
 http://www.rabbitmq.com/plugins.html

> I want to be sure that I have exhausted all possibilities with
> RabbitMQ in our use case to decide if it is something that will work
> in our environment or not, so any insight you have in this area would
> be appreciated.  Perhaps we are attempting to use RabbitMQ in a way
> other than what it was designed for??

RabbitMQ is designed to be used as a buffer. The number of messages that
can be buffered is limited by RAM as well as disk. You will need to
provide enough RAM and disk, in line with the number and size of
messages that need to be buffered.

In your case more RAM will allow you to store more messages. If you are
storing lots of small messages and the message index is the dominating
cost then you might also consider batching messages. This will allow you
to store many more messages.


-Emile