[rabbitmq-discuss] rabbit_persister:extend_transaction in chunks?

Sat Oct 4 05:22:28 BST 2008

Tsuraan,

tsuraan wrote:
>> That would work, but why would you want to do that? Have you obtained
>> some measurements indicating that there is a performance problem in that
>> area?
> 
> No, not a performance issue, but a memory one.

The memory required for the list is linear in the size of the queue and 
independent of the message payload size (because payloads are binaries 
and thus not copied). So normally this shouldn't be an issue. But ...

> I have a file-based queue implementation now,

This changes things since it means the memory footprint of your queues 
is potentially constant rather than O(n), and you want to keep it that 
way. I understand the motivation for your question now.

> it looks like it's easy enough to drop into the
> rabbit_amqqueue_process, with the exception of where it calls
> queue:to_list in purge_message_buffer. I'd like to avoid having to
> have the to_list called, and instead just grab a few (thousand)
> messages at a time and call extend_transaction on the chunks.

It is safe to call extend_transaction and dirty_work (which is what 
actually gets called in the above case) multiple times. Also note that 
you do not need to materialise the entire messages - you just need their 
persistent_keys.

Matthias.