[rabbitmq-discuss] EXT :Re: Memory not flushing to disk

Wed Apr 24 13:50:16 BST 2013

Emile-

Long may be defined as holding tens of millions of messages before the consumers connect to the broker.

You mentioned batching messages.  Is this something that is done by the producer when sending to the broker or a setting within RabbitMQ?

What does RabbitMQ do with the RAM indexes when it is shutdown?  I assume it writes to disk somehow and then reads the indexes back into RAM upon startup.  Does this process only happen upon shutdown and never during execution?  i.e. it never pages the RAM index to disk when out of space?

Logan Rodrian

________________________________________
From: Emile Joubert [emile at rabbitmq.com]
Sent: Wednesday, April 24, 2013 02:48
To: Rodrian, Logan P (IS)
Cc: Discussions about RabbitMQ
Subject: Re: EXT :Re: [rabbitmq-discuss] Memory not flushing to disk

Hi Logan,

On 23/04/13 18:11, Rodrian, Logan P (IS) wrote:

> Let me present our use case in a larger context and see what your
> thoughts are.  Our goal with using RabbitMQ as a broker was to
> provide a managed "disk backup" of data while consumers were
> unavailable, ensuring no data is lost.  In our environment, the
> consumers may not be available for a long time.

Can you place an upper limit on "long"?

> I understand the use of vm_memory_high_watermark to put a "band-aid"
> on the problem, but it sounds like we will still run into issues
> eventually.
>
> So, are there any parameters that can be set on the server to
> configure it for this type of environment?  This includes use of any
> alternative plugins that keep the indexes in a different way and/or
> persist data to the disk using a different approach.

There is at least one alternative message index storage plugin:
 http://hg.rabbitmq.com/rabbitmq-toke/

that uses a driver for Tokyo Cabinet:
 http://hg.rabbitmq.com/toke/

This plugin has not reached official status:
 http://www.rabbitmq.com/plugins.html

> I want to be sure that I have exhausted all possibilities with
> RabbitMQ in our use case to decide if it is something that will work
> in our environment or not, so any insight you have in this area would
> be appreciated.  Perhaps we are attempting to use RabbitMQ in a way
> other than what it was designed for??

RabbitMQ is designed to be used as a buffer. The number of messages that
can be buffered is limited by RAM as well as disk. You will need to
provide enough RAM and disk, in line with the number and size of
messages that need to be buffered.

In your case more RAM will allow you to store more messages. If you are
storing lots of small messages and the message index is the dominating
cost then you might also consider batching messages. This will allow you
to store many more messages.

-Emile