[rabbitmq-discuss] rabbit_persister:extend_transaction in chunks?
matthias at lshift.net
Sat Oct 4 05:22:28 BST 2008
>> That would work, but why would you want to do that? Have you obtained
>> some measurements indicating that there is a performance problem in that
> No, not a performance issue, but a memory one.
The memory required for the list is linear in the size of the queue and
independent of the message payload size (because payloads are binaries
and thus not copied). So normally this shouldn't be an issue. But ...
> I have a file-based queue implementation now,
This changes things since it means the memory footprint of your queues
is potentially constant rather than O(n), and you want to keep it that
way. I understand the motivation for your question now.
> it looks like it's easy enough to drop into the
> rabbit_amqqueue_process, with the exception of where it calls
> queue:to_list in purge_message_buffer. I'd like to avoid having to
> have the to_list called, and instead just grab a few (thousand)
> messages at a time and call extend_transaction on the chunks.
It is safe to call extend_transaction and dirty_work (which is what
actually gets called in the above case) multiple times. Also note that
you do not need to materialise the entire messages - you just need their
More information about the rabbitmq-discuss