Interesting we did not disable it. What is the flip side of doing this?<div><br></div><div>--</div><div>Raphael.<br><br><div class="gmail_quote">On Tue, Oct 25, 2011 at 8:57 AM, Aaron Westendorf <span dir="ltr"><<a href="mailto:aaron@agoragames.com">aaron@agoragames.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Did you disable vm_memory_high_watermark ? �I've attached graphs of<br>
our disk usage before and after we enabled it.<br>
<br>
-Aaron<br>
<div><div></div><div class="h5"><br>
<br>
On Mon, Oct 24, 2011 at 9:07 PM, Raphael Simon <<a href="mailto:raphael@rightscale.com">raphael@rightscale.com</a>> wrote:<br>
> Hello all,<br>
> We are seeing an issue on a production broker where the RabbitMQ process<br>
> writes non stop to files in mnesia/<node>/msg_store_persistent. It keeps<br>
> creating new files and the problem seems to be getting worse. Listing the<br>
> files in that directory shows that it's creating a new 16 MB file every 2 to<br>
> 4 minutes [1].<br>
> The throughput of persistent messages in this broker is orders of magnitude<br>
> less (maybe 20 msg/sec at the most, each being in the 10s of KB) .<br>
> Attaching a shell to the broker process I can retrieve the process status of<br>
> the msg_store_persistent server which is the process causing all the writes,<br>
> not sure what to look for though [2].<br>
> There are about 100 messages sitting in queues on that broker so that should<br>
> not cause that many writes, iostat shows about 6000 writes/s.<br>
> Any pointers to where we should be looking would be greatly appreciated.<br>
> Thanks!<br>
> --<br>
> Raphael.<br>
> [1]<br>
> # ls -alt /mnt/rabbitmq/mnesia/rabbit@broker1-1/msg_store_persistent<br>
> total 453196<br>
> -rw-r--r-- 1 rabbitmq rabbitmq �2973011 2011-10-25 00:55 706.rdq<br>
> drwxr-xr-x 5 rabbitmq rabbitmq � � 4096 2011-10-25 00:54 ..<br>
> drwxr-xr-x 2 rabbitmq rabbitmq � � 4096 2011-10-25 00:53 .<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16779002 2011-10-25 00:49 704.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16779064 2011-10-25 00:44 703.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16778786 2011-10-25 00:39 702.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16780428 2011-10-25 00:33 701.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16779981 2011-10-25 00:28 700.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16779550 2011-10-25 00:24 699.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16780493 2011-10-25 00:21 698.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16777711 2011-10-25 00:17 697.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16782704 2011-10-25 00:13 696.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � � 8991 2011-10-25 00:10 669.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � � 8861 2011-10-25 00:10 667.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �10469 2011-10-25 00:10 665.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �54123 2011-10-25 00:10 662.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �61457 2011-10-25 00:10 658.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � � 6040 2011-10-25 00:10 654.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 13115149 2011-10-25 00:10 634.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16779265 2011-10-25 00:09 695.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16808918 2011-10-25 00:03 694.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16802844 2011-10-24 23:55 690.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16793271 2011-10-24 23:55 689.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16810749 2011-10-24 23:54 688.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16780809 2011-10-24 23:51 683.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16779742 2011-10-24 23:49 682.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16778746 2011-10-24 23:44 681.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16793650 2011-10-24 23:40 680.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16782092 2011-10-24 23:35 679.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16788239 2011-10-24 23:31 678.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16780959 2011-10-24 23:26 677.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16779952 2011-10-24 23:17 675.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16777823 2011-10-24 23:05 672.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16779687 2011-10-24 23:02 671.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 17559554 2011-10-24 20:52 633.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �11847 2011-10-24 20:41 612.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 16100804 2011-10-24 20:41 523.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �11314 2011-10-24 18:39 576.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �28514 2011-10-24 18:16 565.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �17823 2011-10-24 17:24 561.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �14774 2011-10-24 13:25 528.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq 10864262 2011-10-24 13:25 0.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �35908 2011-10-24 13:15 491.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �48383 2011-10-24 13:06 487.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �11912 2011-10-24 12:50 484.rdq<br>
> -rw-r--r-- 1 rabbitmq rabbitmq � �23881 2011-10-24 06:06 437.rdq<br>
> [2]<br>
> {status,<0.237.0>,<br>
> � � � � {module,gen_server2},<br>
> � � � � [[{{#Ref<0.0.3588.227656>,fhc_handle},<br>
> � � � � � �{handle,{file_descriptor,prim_file,{#Port<0.712982>,109}},<br>
> � � � � � � � � � �14469207,14469207,false,0,1048576,[],true,<br>
><br>
> �"/mnt/rabbitmq/mnesia/rabbit@broker1-1/msg_store_persistent/702.rdq",<br>
> � � � � � � � � � �[raw,binary,write],<br>
> � � � � � � � � � �[{write_buffer,1048576}],<br>
> � � � � � � � � � �true,false,<br>
> � � � � � � � � � �{1319,503146,31340}}},<br>
> � � � � � {'$ancestors',[rabbit_sup,<0.73.0>]},<br>
><br>
> {{"/mnt/rabbitmq/mnesia/rabbit@broker1-1/msg_store_persistent/702.rdq",<br>
> � � � � � � fhc_file},<br>
> � � � � � �{file,0,true}},<br>
> � � � � � {fhc_age_tree,{1,<br>
><br>
> �{{1319,503146,31340},#Ref<0.0.3588.227656>,nil,nil}}},<br>
> � � � � � {'$initial_call',{gen,init_it,7}}],<br>
> � � � � �running,<0.74.0>,[],<br>
> � � � � �[{header,"Status for generic server msg_store_persistent"},<br>
> � � � � � {data,[{"Status",running},<br>
> � � � � � � � � �{"Parent",<0.74.0>},<br>
> � � � � � � � � �{"Logged events",[]},<br>
> � � � � � � � � �{"Queued messages",[]}]},<br>
> � � � � � {data,[{"State",<br>
><br>
> {msstate,"/mnt/rabbitmq/mnesia/rabbit@broker1-1/msg_store_persistent",<br>
> � � � � � � � � � � � � � �rabbit_msg_store_ets_index,<br>
> � � � � � � � � � � � � � �{state,262201,<br>
><br>
> "/mnt/rabbitmq/mnesia/rabbit@broker1-1/msg_store_persistent"},<br>
> � � � � � � � � � � � � � �702,#Ref<0.0.3588.227656>,<br>
> � � � � � � � � � � � � � �{dict,0,16,16,8,80,48,<br>
> � � � � � � � � � � � � � � � � �{[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
> � � � � � � � � � � � � � � � � � [],[]},<br>
><br>
> �{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
> � � � � � � � � � � � � � � � � � �[],[]}}},<br>
> � � � � � � � � � � � � � �[],undefined,53527675,424756530,[],<0.240.0>,<br>
> � � � � � � � � � � � � � �266298,258104,270395,<br>
> � � � � � � � � � � � � � �{set,0,16,16,8,80,48,<br>
><br>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
> � � � � � � � � � � � � � � � � �[]},<br>
> � � � � � � � � � � � � � � � � {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
> � � � � � � � � � � � � � � � � � [],[]}}},<br>
> � � � � � � � � � � � � � �{dict,24974,6019,8192,4096,30095,18057,<br>
> � � � � � � � � � � � � � � � � �{[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
> � � � � � � � � � � � � � � � � � [],[]},<br>
><br>
> �{{[[<<109,81,35,172,101,10,100,5,139,235,37,<br>
> � � � � � � � � � � � � � � � � � � � �207,134,159,97,123>>|<br>
> � � � � � � � � � � � � � � � � � � �{#Fun<rabbit_variable_queue.1.360644>,<br>
><br>
> #Fun<rabbit_variable_queue.<a href="tel:20.77393657" value="+12077393657">20.77393657</a>>}],<br>
><br>
> [<<98,60,3,61,105,99,34,128,158,37,55,121,<br>
> � � � � � � � � � � � � � � � � � � � �53,68,253,77>>|<br>
> � � � � � � � � � � � � � � � � � � �{#Fun<rabbit_variable_queue.1.360644>,<br>
><br>
> #Fun<rabbit_variable_queue.<a href="tel:20.77393657" value="+12077393657">20.77393657</a>>}],<br>
> � � � � � � � � � � � � � � � � � � [<<224,253,10,145,132,16,186,61,192,63,<br>
> ..... lots of stuff snipped ....<br>
><br>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
> � � � � � � � � � � � � � � � � � �[],[]},<br>
><br>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
> � � � � � � � � � � � � � � � � � �[],[]}}},<br>
> � � � � � � � � � � � � � �false,16777216,<br>
> � � � � � � � � � � � � � �{dict,0,16,16,8,80,48,<br>
> � � � � � � � � � � � � � � � � �{[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
> � � � � � � � � � � � � � � � � � [],[]},<br>
><br>
> �{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],<br>
> � � � � � � � � � � � � � � � � � �[],[]}}}}}]}]]}<br>
</div></div><div><div></div><div class="h5">> _______________________________________________<br>
> rabbitmq-discuss mailing list<br>
> <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
> <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
><br>
><br>
<br>
<br>
<br>
</div></div><font color="#888888">--<br>
Aaron Westendorf<br>
Senior Software Engineer<br>
Agora Games<br>
359 Broadway<br>
Troy, NY 12180<br>
Phone: <a href="tel:518.268.1000" value="+15182681000">518.268.1000</a><br>
<a href="mailto:aaron@agoragames.com">aaron@agoragames.com</a><br>
<a href="http://www.agoragames.com" target="_blank">www.agoragames.com</a><br>
</font></blockquote></div><br></div>