[rabbitmq-discuss] bug21444 branch experience

Thu Nov 26 16:26:51 GMT 2009

Hello.
We need to handle large queues (50Gb) and thus I started experimenting
with bug21444 branch (revision 7a219f11bb3e) that seems to have this
feature.
First trouble was concerned with leaking disk space. Queue files were
remaining open after deletion.
Simple workaround limited wasted disk space:
src/rabbit_msg_store.erl
@@ -43,7 +43,7 @@

 -define(SERVER, ?MODULE).

--define(MAX_READ_FILE_HANDLES, 256).
+-define(MAX_READ_FILE_HANDLES, 32).
 -define(FILE_SIZE_LIMIT,       (256*1024*1024)).
 -define(SYNC_INTERVAL,         5). %% milliseconds

Another problem arised while starting server with large disk queue. It
seems that erlang gc can't keep up with the speed of reading files and
release unused binary data on time.
Forced garbage collection helped server to survive startup
src/rabbit_msg_store.erl
@@ -790,6 +790,7 @@
     build_index(undefined, Files, [], State).

 build_index(Left, [], FilesToCompact, State) ->
+	[erlang:garbage_collect(P) || P <- erlang:processes()],
     ok = index_delete_by_file(undefined, State),
     Offset = case lists:reverse(index_search_by_file(Left, State)) of
                  [] -> 0;
@@ -801,6 +802,7 @@
             State #msstate { current_file = Left, current_offset = Offset });
 build_index(Left, [File|Files], FilesToCompact,
             State = #msstate { dir = Dir, file_summary = FileSummary }) ->
+	[erlang:garbage_collect(P) || P <- erlang:processes()],
     {ok, Messages} = scan_file_for_valid_messages(Dir, filenum_to_name(File)),
     {ValidMessages, ValidTotalSize, AllValid} =
         lists:foldl(

>From repository changelog I can conclude that there is active
development in branch bug21673 and it have changes concerned with disk
queue. Is it worth trying?

Regards,
Anton Lebedevich.


