[rabbitmq-discuss] RabbitMQ writes non stop to disk

Raphael Simon raphael at rightscale.com
Tue Oct 25 17:26:54 BST 2011


Interesting we did not disable it. What is the flip side of doing this?

--
Raphael.

On Tue, Oct 25, 2011 at 8:57 AM, Aaron Westendorf <aaron at agoragames.com>wrote:

> Did you disable vm_memory_high_watermark ?  I've attached graphs of
> our disk usage before and after we enabled it.
>
> -Aaron
>
>
> On Mon, Oct 24, 2011 at 9:07 PM, Raphael Simon <raphael at rightscale.com>
> wrote:
> > Hello all,
> > We are seeing an issue on a production broker where the RabbitMQ process
> > writes non stop to files in mnesia/<node>/msg_store_persistent. It keeps
> > creating new files and the problem seems to be getting worse. Listing the
> > files in that directory shows that it's creating a new 16 MB file every
> 2 to
> > 4 minutes [1].
> > The throughput of persistent messages in this broker is orders of
> magnitude
> > less (maybe 20 msg/sec at the most, each being in the 10s of KB) .
> > Attaching a shell to the broker process I can retrieve the process
> status of
> > the msg_store_persistent server which is the process causing all the
> writes,
> > not sure what to look for though [2].
> > There are about 100 messages sitting in queues on that broker so that
> should
> > not cause that many writes, iostat shows about 6000 writes/s.
> > Any pointers to where we should be looking would be greatly appreciated.
> > Thanks!
> > --
> > Raphael.
> > [1]
> > # ls -alt /mnt/rabbitmq/mnesia/rabbit at broker1-1/msg_store_persistent
> > total 453196
> > -rw-r--r-- 1 rabbitmq rabbitmq  2973011 2011-10-25 00:55 706.rdq
> > drwxr-xr-x 5 rabbitmq rabbitmq     4096 2011-10-25 00:54 ..
> > drwxr-xr-x 2 rabbitmq rabbitmq     4096 2011-10-25 00:53 .
> > -rw-r--r-- 1 rabbitmq rabbitmq 16779002 2011-10-25 00:49 704.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16779064 2011-10-25 00:44 703.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16778786 2011-10-25 00:39 702.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16780428 2011-10-25 00:33 701.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16779981 2011-10-25 00:28 700.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16779550 2011-10-25 00:24 699.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16780493 2011-10-25 00:21 698.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16777711 2011-10-25 00:17 697.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16782704 2011-10-25 00:13 696.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq     8991 2011-10-25 00:10 669.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq     8861 2011-10-25 00:10 667.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    10469 2011-10-25 00:10 665.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    54123 2011-10-25 00:10 662.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    61457 2011-10-25 00:10 658.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq     6040 2011-10-25 00:10 654.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 13115149 2011-10-25 00:10 634.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16779265 2011-10-25 00:09 695.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16808918 2011-10-25 00:03 694.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16802844 2011-10-24 23:55 690.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16793271 2011-10-24 23:55 689.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16810749 2011-10-24 23:54 688.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16780809 2011-10-24 23:51 683.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16779742 2011-10-24 23:49 682.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16778746 2011-10-24 23:44 681.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16793650 2011-10-24 23:40 680.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16782092 2011-10-24 23:35 679.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16788239 2011-10-24 23:31 678.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16780959 2011-10-24 23:26 677.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16779952 2011-10-24 23:17 675.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16777823 2011-10-24 23:05 672.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16779687 2011-10-24 23:02 671.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 17559554 2011-10-24 20:52 633.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    11847 2011-10-24 20:41 612.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 16100804 2011-10-24 20:41 523.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    11314 2011-10-24 18:39 576.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    28514 2011-10-24 18:16 565.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    17823 2011-10-24 17:24 561.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    14774 2011-10-24 13:25 528.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq 10864262 2011-10-24 13:25 0.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    35908 2011-10-24 13:15 491.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    48383 2011-10-24 13:06 487.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    11912 2011-10-24 12:50 484.rdq
> > -rw-r--r-- 1 rabbitmq rabbitmq    23881 2011-10-24 06:06 437.rdq
> > [2]
> > {status,<0.237.0>,
> >         {module,gen_server2},
> >         [[{{#Ref<0.0.3588.227656>,fhc_handle},
> >            {handle,{file_descriptor,prim_file,{#Port<0.712982>,109}},
> >                    14469207,14469207,false,0,1048576,[],true,
> >
> >  "/mnt/rabbitmq/mnesia/rabbit at broker1-1/msg_store_persistent/702.rdq",
> >                    [raw,binary,write],
> >                    [{write_buffer,1048576}],
> >                    true,false,
> >                    {1319,503146,31340}}},
> >           {'$ancestors',[rabbit_sup,<0.73.0>]},
> >
> > {{"/mnt/rabbitmq/mnesia/rabbit at broker1-1/msg_store_persistent/702.rdq",
> >             fhc_file},
> >            {file,0,true}},
> >           {fhc_age_tree,{1,
> >
> >  {{1319,503146,31340},#Ref<0.0.3588.227656>,nil,nil}}},
> >           {'$initial_call',{gen,init_it,7}}],
> >          running,<0.74.0>,[],
> >          [{header,"Status for generic server msg_store_persistent"},
> >           {data,[{"Status",running},
> >                  {"Parent",<0.74.0>},
> >                  {"Logged events",[]},
> >                  {"Queued messages",[]}]},
> >           {data,[{"State",
> >
> > {msstate,"/mnt/rabbitmq/mnesia/rabbit at broker1-1/msg_store_persistent",
> >                            rabbit_msg_store_ets_index,
> >                            {state,262201,
> >
> > "/mnt/rabbitmq/mnesia/rabbit at broker1-1/msg_store_persistent"},
> >                            702,#Ref<0.0.3588.227656>,
> >                            {dict,0,16,16,8,80,48,
> >
>  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> >                                   [],[]},
> >
> >  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> >                                    [],[]}}},
> >                            [],undefined,53527675,424756530,[],<0.240.0>,
> >                            266298,258104,270395,
> >                            {set,0,16,16,8,80,48,
> >
> > {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> >                                  []},
> >
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> >                                   [],[]}}},
> >                            {dict,24974,6019,8192,4096,30095,18057,
> >
>  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> >                                   [],[]},
> >
> >  {{[[<<109,81,35,172,101,10,100,5,139,235,37,
> >                                        207,134,159,97,123>>|
> >
>  {#Fun<rabbit_variable_queue.1.360644>,
> >
> > #Fun<rabbit_variable_queue.20.77393657>}],
> >
> > [<<98,60,3,61,105,99,34,128,158,37,55,121,
> >                                        53,68,253,77>>|
> >
>  {#Fun<rabbit_variable_queue.1.360644>,
> >
> > #Fun<rabbit_variable_queue.20.77393657>}],
> >
> [<<224,253,10,145,132,16,186,61,192,63,
> > ..... lots of stuff snipped ....
> >
> > {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> >                                    [],[]},
> >
> > {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> >                                    [],[]}}},
> >                            false,16777216,
> >                            {dict,0,16,16,8,80,48,
> >
>  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> >                                   [],[]},
> >
> >  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> >                                    [],[]}}}}}]}]]}
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> >
>
>
>
> --
> Aaron Westendorf
> Senior Software Engineer
> Agora Games
> 359 Broadway
> Troy, NY 12180
> Phone: 518.268.1000
> aaron at agoragames.com
> www.agoragames.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111025/82fbd4df/attachment.htm>


More information about the rabbitmq-discuss mailing list