[rabbitmq-discuss] RabbitMQ crashed

[Kapil Goyal]

Hi,

 

I had a RabbitMQ instance crash under constant load. I was sending about
400 messages per second to a queue and the queue size was growing with
time, i.e. the draining of queue was slower than the rate of publishing.
Below is the error from the logs. Can you help me understand the root
cause of this crash?

 

=ERROR REPORT==== 21-Oct-2013::20:32:11 ===
** Generic server <0.764.0> terminating
** Last message in was pre_hibernate
** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,1,<0.757.0>,
                         <0.762.0>,<0.757.0>,
                         <<"127.0.0.1:42600 -> 127.0.0.1:5672">>,
                         {lstate,<0.763.0>,true,false},
                         none,8434,
                         {[{8433,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8432}},
                           {8432,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8431}},
                           {8431,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8430}},
                           {8430,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8429}},
                           {8429,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8428}},
                           {8428,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8427}},
                           {8427,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8426}},
                           {8426,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8425}},
                           {8425,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8424}},
                           {8424,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8423}},
                           {8423,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8422}},
                           {8422,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8421}},
                           {8421,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8420}},
                           {8420,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8419}},
                           {8419,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8418}},
                           {8418,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8417}},
                           {8417,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8416}},
                           {8416,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8415}},
                           {8415,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8414}},
                           {8414,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8413}}],
                          [{8413,<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,
                            {<0.233.0>,8412}}]},
                         {user,<<"vsm">>,
                          [administrator],
                          rabbit_auth_backend_internal,
                          {internal_user,<<"vsm">>,
 
<<110,220,216,189,136,246,198,45,125,163,135,132,58,
                             245,179,46,244,240,41,150>>,
                           [administrator]}},
                         <<"vshield">>,<<"vsm_request_queue">>,
                         {dict,1,16,16,8,80,48,
 
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],
                            [[<0.233.0>|
                              {resource,<<"vshield">>,queue,
                               <<"vsm_request_queue">>}]],
                            [],[],[],[],[],[]}}},
                         {dict,1,16,16,8,80,48,
 
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],
                            [[<0.233.0>|#Ref<0.0.0.21445>]],
                            [],[],[],[],[],[]}}},
                         {dict,1,16,16,8,80,48,
 
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],
                            [[<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>|
                              {amqqueue,
                               {resource,<<"vshield">>,queue,
                                <<"vsm_request_queue">>},
                               true,false,none,
                               [{<<"x-message-ttl">>,long,600000}],
                               <0.233.0>,[],[],undefined,[]}]],
                            [],[],[],[],[],[],[]}}},
                         {set,0,16,16,8,80,48,
 
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
 
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,1,16,16,8,80,48,
 
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],
                            [[<0.233.0>|
                              {1,
                               {<<"amq.ctag-8UDwLESsF_jg_1qLyYfVIA">>,nil,
                                nil}}]],
                            [],[],[],[],[],[]}}},
                         {set,1,16,16,8,80,48,
 
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],
                            [<0.233.0>],
                            [],[],[],[],[],[]}}},
                         <0.755.0>,
                         {state,fine,5000,#Ref<0.0.0.231446>},
                         false,1,
                         {{0,nil},{0,nil}},
                         [],
                         [{<<"exchange_exchange_bindings">>,bool,true},
                          {<<"consumer_cancel_notify">>,bool,true},
                          {<<"basic.nack">>,bool,true},
                          {<<"publisher_confirms">>,bool,true}],
                         none}
** Reason for termination ==
** {timeout,{gen_server,call,[<0.763.0>,get_prefetch_limit]}}

 

Thanks

Kapil

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131023/af70abb1/attachment.htm>