[rabbitmq-discuss] RabbitMQ crashed in ets:insert_new - looks like a genuine bug...

[Eugene Kirpichov]

A lot of clients (a thousand or more) were rapidly publishing 1kb
messages to a queue, and then rabbitmq crashed.

In fact I had a cluster of 4 rabbits, and 2 of them crashed as a
result. The remaining 2 continued working ok.

Here's a crash report from rabbit-sasl.log. I do not give the full log
because it's large, contains message data (which my employer might not
like) and I'm too lazy to automatically snip it.
But the log is really full of things exactly like what I show. This
exact message gets repeated many times in the same second, and then it
finally crashed.

What other information can I provide to resolve this? Could this be an
error on my, not rabbit's, part? Having a sudden rabbitmq crash is not
really what I'd like to have in production :-|

=CRASH REPORT==== 11-Aug-2011::17:56:19 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.16624.0>
    registered_name: []
    exception exit: {badarg,
                        [{ets,insert_new,
                             [303172,
                              {<<223,221,16,201,23,190,196,251,169,11,157,145,
                                 94,36,1,105>>,
                               {basic_message,
                                   {resource,<<"/">>,exchange,<<>>},

[<<"results-8808E5FBBC714C9E880F9FD30F443151.TestApp.rmq002">>],
                                   {content,60,none,
                                       <<....>>, % (snipped)
                                       rabbit_framing_amqp_0_9_1,
                                       [<<....>>]}, % (snipped too)
                                   <<223,221,16,201,23,190,196,251,169,11,157,
                                     145,94,36,1,105>>,
                                   true},
                               1}]},
                         {rabbit_msg_store,update_msg_cache,3},
                         {rabbit_msg_store,write,3},
                         {rabbit_variable_queue,
                             '-with_immutable_msg_store_state/3-fun-0-',2},
                         {rabbit_variable_queue,with_msg_store_state,3},
                         {rabbit_variable_queue,
                             with_immutable_msg_store_state,3},
                         {rabbit_variable_queue,maybe_write_msg_to_disk,3},
                         {rabbit_variable_queue,maybe_write_to_disk,4}]}
      in function  gen_server2:terminate/3
    ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.137.0>]
    messages: [{'$gen_cast',{ack,none,[46689],<0.16623.0>}},
                  {'$gen_cast',{ack,none,[46690],<0.16623.0>}}]
    links: [<0.263.0>]
    dictionary: [{fhc_age_tree,{0,nil}},
                  {{ch,<0.16623.0>},
                   {cr,1,<0.16623.0>,<0.16628.0>,#Ref<0.0.0.16807>,
                       {set,2,16,16,8,80,48,
                            {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                            {{[],[],[],[],[],[],[],[],
                              [46690],
                              [],[],[],[],
                              [46689],
                              [],[]}}},
                       false,none,0}},
                  {guid,{{7,<0.16624.0>},1}}]
    trap_exit: true
    status: running
    heap_size: 1682835
    stack_size: 24
    reductions: 1260360700
  neighbours:


-- 
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/