[rabbitmq-discuss] Strange failures (Generic server <xxx> terminating) from RabbitMQ

Matt Pietrek mpietrek at skytap.com
Wed Jul 17 00:26:54 BST 2013


We're seeing a bunch of failure messages from our Ruby clients to RabbitMQ
3.0.2. From the client side, we see 541 "Internal Error"

On the server side logs, we see a bunch of errors. Normally I can at least
hazard a guess what's wrong, but I have no idea with this:


=ERROR REPORT==== 16-Jul-2013::22:43:15 ===
** Generic server <0.4358.0> terminating
** Last message in was {can_send,<0.4355.0>,true}
** When Server state == {lim,1,<0.4359.0>,false,[],1}
** Reason for termination ==
** {function_clause,[{orddict,update,

[<0.4355.0>,#Fun<rabbit_limiter.3.113433752>,[]],
                              [{file,"orddict.erl"},{line,153}]},
                     {rabbit_limiter,limit_queue,2,[]},
                     {rabbit_limiter,handle_call,3,[]},
                     {gen_server2,handle_msg,2,[]},
                     {proc_lib,init_p_do_apply,3,
                               [{file,"proc_lib.erl"},{line,227}]}]}



Or like this:
--------------------


=ERROR REPORT==== 16-Jul-2013::22:44:15 ===
** Generic server <0.4563.0> terminating
** Last message in was {'$gen_cast',
                           {method,
                               {'basic.consume',0,<<"web-tasks-mpietrek">>,

<<"web-tasks-mpietrek-1374014655000-264079359112">>,
                                   false,false,false,false,[]},
                               none,noflow}}
** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,41,<0.4540.0>,
                         <0.4560.0>,<0.4540.0>,
                         <<"192.168.78.13:57671 -> 192.168.78.50:5672">>,
                         {lstate,<0.4561.0>,true,false},
                         none,2,
                         {[],[]},
                         {user,<<"guest">>,
                          [administrator],
                          rabbit_auth_backend_internal,
                          {internal_user,<<"guest">>,

<<78,56,17,4,226,151,136,14,156,84,34,25,101,29,104,
                             78,125,130,202,207>>,
                           [administrator]}},
                         <<"/">>,<<"web-tasks-mpietrek">>,
                         {dict,1,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],
                            [[<0.4558.0>|
                              {resource,<<"/">>,queue,
                               <<"web-tasks-mpietrek">>}]],
                            [],[],[],[],[],[],[],[]}}},
                         {dict,3,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],
                            [[<0.4558.0>|#Ref<0.0.0.83887>]],
                            [],[],[],[],
                            [[<2783.2507.0>|#Ref<0.0.0.83888>]],
                            [[<2782.2563.0>|#Ref<0.0.0.83889>]],
                            [],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {set,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         {set,1,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],
                            [<0.4558.0>],
                            [],[],[],[],[],[],[],[]}}},
                         <0.4539.0>,
                         {state,fine,5000,#Ref<0.0.0.86733>},
                         false,1,
                         {{0,nil},{0,nil}},
                         [],
                         [{<<"consumer_cancel_notify">>,bool,true}],
                         none}

Or like this:
--------------------

=ERROR REPORT==== 16-Jul-2013::22:44:15 ===
** Generic server <0.4561.0> terminating
** Last message in was {can_send,<0.4558.0>,true}
** When Server state == {lim,1,<0.4563.0>,false,[],1}
** Reason for termination ==
** {function_clause,[{orddict,update,

[<0.4558.0>,#Fun<rabbit_limiter.3.113433752>,[]],
                              [{file,"orddict.erl"},{line,153}]},
                     {rabbit_limiter,limit_queue,2,[]},
                     {rabbit_limiter,handle_call,3,[]},
                     {gen_server2,handle_msg,2,[]},
                     {proc_lib,init_p_do_apply,3,
                               [{file,"proc_lib.erl"},{line,227}]}]}

=ERROR REPORT==== 16-Jul-2013::22:44:15 ===
** Generic server <0.4558.0> terminating
** Last message in was {basic_consume,false,<0.4563.0>,<0.4561.0>,true,

<<"web-tasks-mpietrek-1374014655000-264079359112">>,
                           false,none,
                           {'basic.consume_ok',

<<"web-tasks-mpietrek-1374014655000-264079359112">>}}
** When Server state == {q,
                         {amqqueue,
                          {resource,<<"/">>,queue,<<"web-tasks-mpietrek">>},
                          true,false,none,[],<0.4558.0>,[],[],
                          [{vhost,<<"/">>},
                           {name,<<"ha-web-tasks-mpietrek">>},
                           {pattern,<<"web-tasks-mpietrek">>},
                           {definition,[{<<"ha-mode">>,<<"all">>}]},
                           {priority,0}],
                          []},
                         none,true,rabbit_mirror_queue_master,
                         {state,
                          {resource,<<"/">>,queue,<<"web-tasks-mpietrek">>},
                          <0.4566.0>,<0.4565.0>,rabbit_variable_queue,
                          {vqstate,
                           {0,{[],[]}},
                           {0,{[],[]}},
                           {delta,undefined,0,undefined},
                           {0,{[],[]}},
                           {13,
                            {[{msg_status,12,

<<114,86,130,79,196,29,163,190,216,213,230,38,
                                 174,139,39,60>>,
                               {basic_message,
                                {resource,<<"/">>,exchange,<<"skytap">>},
                                [<<"web-tasks-mpietrek.mpietrek.default">>],
                                {content,60,
                                 {'P_basic',<<"application/octet-stream">>,

undefined,undefined,2,0,undefined,undefined,
                                  undefined,undefined,undefined,undefined,
                                  undefined,undefined,undefined},

<<152,0,24,97,112,112,108,105,99,97,116,105,
                                   111,110,47,111,99,116,101,116,45,115,116,
                                   114,101,97,109,2,0>>,
                                 rabbit_framing_amqp_0_9_1,

[<<"{\"payload\":{\"object\":\"BAhvOh1EZWxldGVFeHBpcmVkVXBsb2Fkc1Rhc2sGOhJAZ3JhY2VfcGVyaW9k\\nbzocQWN0aXZlU3VwcG9ydDo6RHVyYXRpb24HOgtAdmFsdWVpAwB1EjoLQHBh\\ncnRzWwZbBzoJZGF5c2kT\\n\"},\"_serialization_type\":\"marshal\",\"_timestamp\":\"2013-07-16T22:43:55Z\",\"_message_type\":\"request\",\"_unique_id\":\"app1:req:wfe-em-init.80690eb0d0940130f02f000c291bd1d8.2.372:336\",\"caller_context\":\"wfe-em-init.80690eb0d0940130f02f000c291bd1d8.2.373\",\"action\":\"async-task\",\"_mq_protocol_version\":1}">>]},

<<114,86,130,79,196,29,163,190,216,213,230,38,
                                  174,139,39,60>>,
                                true},
                               true,false,true,true,
                               {message_properties,undefined,false}},
                              {msg_status,11,

<<12,29,215,111,220,221,165,115,106,150,132,130,
                                 118,94,193,26>>,
                               {basic_message,
                                {resource,<<"/">>,exchange,<<"skytap">>},
                                [<<"web-tasks-mpietrek.mpietrek.default">>],
                                {content,60,
                                 {'P_basic',<<"application/octet-stream">>,

undefined,undefined,2,0,undefined,undefined,
                                  undefined,undefined,undefined,undefined,
                                  undefined,undefined,undefined},

<<152,0,24,97,112,112,108,105,99,97,116,105,
                                   111,110,47,111,99,116,101,116,45,115,116,
                                   114,101,97,109,2,0>>,
                                 rabbit_framing_amqp_0_9_1,

<rest of error omitted>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130716/7f373738/attachment.htm>


More information about the rabbitmq-discuss mailing list