[rabbitmq-discuss] Persistent queues

[Simon MacMullen]

On 07/06/13 15:12, Marc Labbe wrote:
> Hi guys,

Hi.

> From what I (believe I) saw and read, I really expected Rabbit to route
> messages and physically save them to disk for each of the consumers to
> read (1 queue/consumer, 1msg copy/queue). The reason I expected that is
> because if messages are ack'ed by consumers 1 and 2, I expected their
> queue to be free of their messages while consumer 3 would still need
> them to be persisted until it has read them. That was possible a false
> assumption I should have verified with you guys. It would indeed be more
> optimal to store a single copy of the messages and only keep the
> metadata. *Is this correct?*

Yes. The message (properties and body) is persisted once for all queues; 
the metadata (location in queue and whether the message is out for 
delivery) is duplicated across all queues.

(Minor exception: dead-lettering a message creates a new copy, breaking 
this property.)

> Also, during a test where I continuously sent (a lot) of messages while
> my big consumer 3 was offline. The observation I made then is that
> _obviously:_
> - data was accumulating on disk
> - memory usage on all nodes went up. to the point where it eventually
> choked the brokers
>
> I assumed then that the increasing memory usage was because the queue
> and msg metadata was increasing in Mnesia. This was a decisive factor to
> me because I didn't want to be in a situation where my realtime
> consumers would be affected by a slow moving elephant. Is there way I
> could have avoided that situation I didn't see?

Technically queue contents are not held in Mnesia but that's a side issue.

The default message store index has a per-message in-memory cost of ~160 
bytes (again, not per queue). The message store index is pluggable and 
can be replaced, there's an implementation based on Tokyo Cabinet which 
has no memory cost per message. But it's not part of the default 
distribution of RabbitMQ making it rather fiddlier to install.

If you are prepared to lose the messages that are unconsumed by your 
elephant, http://www.rabbitmq.com/ttl.html might be another answer.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, Pivotal