Hello all,<div><br></div><div>We are seeing an issue on a production broker where the RabbitMQ process writes non stop to files in mnesia/<node>/msg_store_persistent. It keeps creating new files and the problem seems to be getting worse. Listing the files in that directory shows that it's creating a new 16 MB file every 2 to 4 minutes [1].</div>
<div><br></div><div>The throughput of persistent messages in this broker is orders of magnitude less (maybe 20 msg/sec at the most, each being in the 10s of KB) .</div><div><br></div><div>Attaching a shell to the broker process I can retrieve the process status of the msg_store_persistent server which is the process causing all the writes, not sure what to look for though [2].</div>
<div><br></div><div>There are about 100 messages sitting in queues on that broker so that should not cause that many writes, iostat shows about 6000 writes/s.</div><div><br></div><div>Any pointers to where we should be looking would be greatly appreciated.</div>
<div><br></div><div>Thanks!</div><div><br></div><div>--</div><div>Raphael.</div><div><br></div><div>[1]</div><div><div><div><font class="Apple-style-span" face="'courier new', monospace"># ls -alt /mnt/rabbitmq/mnesia/rabbit@broker1-1/msg_store_persistent</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">total 453196</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq �2973011 2011-10-25 00:55 706.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">drwxr-xr-x 5 rabbitmq rabbitmq � � 4096 2011-10-25 00:54 ..</font></div><div><font class="Apple-style-span" face="'courier new', monospace">drwxr-xr-x 2 rabbitmq rabbitmq � � 4096 2011-10-25 00:53 .</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16779002 2011-10-25 00:49 704.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16779064 2011-10-25 00:44 703.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16778786 2011-10-25 00:39 702.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16780428 2011-10-25 00:33 701.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16779981 2011-10-25 00:28 700.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16779550 2011-10-25 00:24 699.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16780493 2011-10-25 00:21 698.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16777711 2011-10-25 00:17 697.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16782704 2011-10-25 00:13 696.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � � 8991 2011-10-25 00:10 669.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � � 8861 2011-10-25 00:10 667.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �10469 2011-10-25 00:10 665.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �54123 2011-10-25 00:10 662.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �61457 2011-10-25 00:10 658.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � � 6040 2011-10-25 00:10 654.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 13115149 2011-10-25 00:10 634.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16779265 2011-10-25 00:09 695.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16808918 2011-10-25 00:03 694.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16802844 2011-10-24 23:55 690.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16793271 2011-10-24 23:55 689.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16810749 2011-10-24 23:54 688.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16780809 2011-10-24 23:51 683.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16779742 2011-10-24 23:49 682.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16778746 2011-10-24 23:44 681.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16793650 2011-10-24 23:40 680.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16782092 2011-10-24 23:35 679.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16788239 2011-10-24 23:31 678.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16780959 2011-10-24 23:26 677.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16779952 2011-10-24 23:17 675.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16777823 2011-10-24 23:05 672.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16779687 2011-10-24 23:02 671.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 17559554 2011-10-24 20:52 633.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �11847 2011-10-24 20:41 612.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 16100804 2011-10-24 20:41 523.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �11314 2011-10-24 18:39 576.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �28514 2011-10-24 18:16 565.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �17823 2011-10-24 17:24 561.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �14774 2011-10-24 13:25 528.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq 10864262 2011-10-24 13:25 0.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �35908 2011-10-24 13:15 491.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �48383 2011-10-24 13:06 487.rdq</font></div><div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �11912 2011-10-24 12:50 484.rdq</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">-rw-r--r-- 1 rabbitmq rabbitmq � �23881 2011-10-24 06:06 437.rdq</font></div></div></div><div><br></div><div>[2]</div><div><div><font class="Apple-style-span" face="'courier new', monospace">{status,<0.237.0>,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � {module,gen_server2},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � [[{{#Ref<0.0.3588.227656>,fhc_handle},</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � �{handle,{file_descriptor,prim_file,{#Port<0.712982>,109}},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � �14469207,14469207,false,0,1048576,[],true,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � �"/mnt/rabbitmq/mnesia/rabbit@broker1-1/msg_store_persistent/702.rdq",</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � �[raw,binary,write],</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � �[{write_buffer,1048576}],</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � �true,false,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � �{1319,503146,31340}}},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � {'$ancestors',[rabbit_sup,<0.73.0>]},</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � {{"/mnt/rabbitmq/mnesia/rabbit@broker1-1/msg_store_persistent/702.rdq",</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � fhc_file},</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � �{file,0,true}},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � {fhc_age_tree,{1,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � �{{1319,503146,31340},#Ref<0.0.3588.227656>,nil,nil}}},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � {'$initial_call',{gen,init_it,7}}],</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � �running,<0.74.0>,[],</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � �[{header,"Status for generic server msg_store_persistent"},</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � {data,[{"Status",running},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � �{"Parent",<0.74.0>},</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � �{"Logged events",[]},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � �{"Queued messages",[]}]},</font></div>
</div><div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � {data,[{"State",</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � {msstate,"/mnt/rabbitmq/mnesia/rabbit@broker1-1/msg_store_persistent",</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � �rabbit_msg_store_ets_index,</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � �{state,262201,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � "/mnt/rabbitmq/mnesia/rabbit@broker1-1/msg_store_persistent"},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � �702,#Ref<0.0.3588.227656>,</font></div>
<div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � �{dict,0,16,16,8,80,48,</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � �{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � [],[]},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � �{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � �[],[]}}},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � �[],undefined,53527675,424756530,[],<0.240.0>,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � �266298,258104,270395,</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � �{set,0,16,16,8,80,48,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � �[]},</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � [],[]}}},</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � �{dict,24974,6019,8192,4096,30095,18057,</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � �{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � [],[]},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � �{{[[<<109,81,35,172,101,10,100,5,139,235,37,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � � � �207,134,159,97,123>>|</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � � �{#Fun<rabbit_variable_queue.1.360644>,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � � � #Fun<rabbit_variable_queue.20.77393657>}],</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � � [<<98,60,3,61,105,99,34,128,158,37,55,121,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � � � �53,68,253,77>>|</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � � �{#Fun<rabbit_variable_queue.1.360644>,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � � � #Fun<rabbit_variable_queue.20.77393657>}],</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � � [<<224,253,10,145,132,16,186,61,192,63,</font></div>
</div></div><div><font class="Apple-style-span" face="'courier new', monospace">..... lots of stuff snipped ....</font></div><div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � {[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � �[],[]},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � {[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � �[],[]}}},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � �false,16777216,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � �{dict,0,16,16,8,80,48,</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � �{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � [],[]},</font></div><div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � �{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">� � � � � � � � � � � � � � � � � �[],[]}}}}}]}]]}</font></div></div>