<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
span.EmailStyle17
        {mso-style-type:personal-compose;
        font-family:"Calibri","sans-serif";
        color:windowtext;}
.MsoChpDefault
        {mso-style-type:export-only;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I had a RabbitMQ instance crash under constant load. I was sending about 400 messages per second to a queue and the queue size was growing with time, i.e. the draining of queue was slower than the rate of publishing. Below is the error from the logs. Can you help me understand the root cause of this crash?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";color:#333333;background:white'>=ERROR REPORT==== 21-Oct-2013::20:32:11 ===</span><span style='font-size:10.0pt;font-family:"Courier New";color:#333333'><br><span style='background:white'>** Generic server <0.764.0> terminating</span><br><span style='background:white'>** Last message in was pre_hibernate</span><br><span style='background:white'>** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,1,<0.757.0>,</span><br><span style='background:white'> <0.762.0>,<0.757.0>,</span><br><span style='background:white'> <<"127.0.0.1:42600 -> 127.0.0.1:5672">>,</span><br><span style='background:white'> {lstate,<0.763.0>,true,false},</span><br><span style='background:white'> none,8434,</span><br><span style='background:white'> {[{8433,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8432}},</span><br><span style='background:white'> {8432,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8431}},</span><br><span style='background:white'> {8431,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8430}},</span><br><span style='background:white'> {8430,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8429}},</span><br><span style='background:white'> {8429,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8428}},</span><br><span style='background:white'> {8428,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8427}},</span><br><span style='background:white'> {8427,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8426}},</span><br><span style='background:white'> {8426,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8425}},</span><br><span style='background:white'> {8425,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8424}},</span><br><span style='background:white'> {8424,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8423}},</span><br><span style='background:white'> {8423,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8422}},</span><br><span style='background:white'> {8422,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8421}},</span><br><span style='background:white'> {8421,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8420}},</span><br><span style='background:white'> {8420,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8419}},</span><br><span style='background:white'> {8419,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8418}},</span><br><span style='background:white'> {8418,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8417}},</span><br><span style='background:white'> {8417,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8416}},</span><br><span style='background:white'> {8416,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8415}},</span><br><span style='background:white'> {8415,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8414}},</span><br><span style='background:white'> {8414,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8413}}],</span><br><span style='background:white'> [{8413,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,</span><br><span style='background:white'> {<0.233.0>,8412}}]},</span><br><span style='background:white'> {user,<<"vsm">>,</span><br><span style='background:white'> [administrator],</span><br><span style='background:white'> rabbit_auth_backend_internal,</span><br><span style='background:white'> {internal_user,<<"vsm">>,</span><br><span style='background:white'> <<110,220,216,189,136,246,198,45,125,163,135,132,58,</span><br><span style='background:white'> 245,179,46,244,240,41,150>>,</span><br><span style='background:white'> [administrator]}},</span><br><span style='background:white'> <<"vshield">>,<<"vsm_request_queue">>,</span><br><span style='background:white'> {dict,1,16,16,8,80,48,</span><br><span style='background:white'> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</span><br><span style='background:white'> {{[],[],[],[],[],[],[],[],[],</span><br><span style='background:white'> [[<0.233.0>|</span><br><span style='background:white'> {resource,<<"vshield">>,queue,</span><br><span style='background:white'> <<"vsm_request_queue">>}]],</span><br><span style='background:white'> [],[],[],[],[],[]}}},</span><br><span style='background:white'> {dict,1,16,16,8,80,48,</span><br><span style='background:white'> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</span><br><span style='background:white'> {{[],[],[],[],[],[],[],[],[],</span><br><span style='background:white'> [[<0.233.0>|#Ref<0.0.0.21445>]],</span><br><span style='background:white'> [],[],[],[],[],[]}}},</span><br><span style='background:white'> {dict,1,16,16,8,80,48,</span><br><span style='background:white'> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</span><br><span style='background:white'> {{[],[],[],[],[],[],[],[],</span><br><span style='background:white'> [[<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>|</span><br><span style='background:white'> {amqqueue,</span><br><span style='background:white'> {resource,<<"vshield">>,queue,</span><br><span style='background:white'> <<"vsm_request_queue">>},</span><br><span style='background:white'> true,false,none,</span><br><span style='background:white'> [{<<"x-message-ttl">>,long,600000}],</span><br><span style='background:white'> <0.233.0>,[],[],undefined,[]}]],</span><br><span style='background:white'> [],[],[],[],[],[],[]}}},</span><br><span style='background:white'> {set,0,16,16,8,80,48,</span><br><span style='background:white'> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</span><br><span style='background:white'> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},</span><br><span style='background:white'> {dict,1,16,16,8,80,48,</span><br><span style='background:white'> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</span><br><span style='background:white'> {{[],[],[],[],[],[],[],[],[],</span><br><span style='background:white'> [[<0.233.0>|</span><br><span style='background:white'> {1,</span><br><span style='background:white'> {<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,nil,</span><br><span style='background:white'> nil}}]],</span><br><span style='background:white'> [],[],[],[],[],[]}}},</span><br><span style='background:white'> {set,1,16,16,8,80,48,</span><br><span style='background:white'> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</span><br><span style='background:white'> {{[],[],[],[],[],[],[],[],[],</span><br><span style='background:white'> [<0.233.0>],</span><br><span style='background:white'> [],[],[],[],[],[]}}},</span><br><span style='background:white'> <0.755.0>,</span><br><span style='background:white'> {state,fine,5000,#Ref<0.0.0.231446>},</span><br><span style='background:white'> false,1,</span><br><span style='background:white'> {{0,nil},{0,nil}},</span><br><span style='background:white'> [],</span><br><span style='background:white'> [{<<"exchange_exchange_bindings">>,bool,true},</span><br><span style='background:white'> {<<"consumer_cancel_notify">>,bool,true},</span><br><span style='background:white'> {<<"basic.nack">>,bool,true},</span><br><span style='background:white'> {<<"publisher_confirms">>,bool,true}],</span><br><span style='background:white'> none}</span><br><span style='background:white'>** Reason for termination ==</span><br><span style='background:white'>** {timeout,{gen_server,call,[<0.763.0>,get_prefetch_limit]}}</span></span><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks<o:p></o:p></p><p class=MsoNormal>Kapil<o:p></o:p></p></div></body></html>