[rabbitmq-discuss] broker goes down after many new connections started

[Brian Whitman]

We have a series of machines that we boot at once that all consume and
produce messages on about 20 different queues all to the same broker. The
broker has been terminating when these machines boot recently. We're not
sure why. Here's some log data:
=ERROR REPORT==== 16-Sep-2009::11:48:52 ===
** Generic server <0.2390.0> terminating
** Last message in was {inet_async,#Port<0.3893>,173,{ok,#Port<0.430698>}}
** When Server state == none
** Reason for termination ==
** {cannot_accept,{error,emfile}}


=ERROR REPORT==== 16-Sep-2009::11:48:52 ===
** gen_event handler rabbit_sasl_report_file_h crashed.
** Was installed in error_logger
** Last event was: {error_report,<0.101.0>,
                       {<0.2390.0>,crash_report,

 [[{initial_call,{tcp_acceptor,init,['Argument__1']}},
                          {pid,<0.2390.0>},
                          {registered_name,[]},
                          {error_info,
                              {exit,
                                  {cannot_accept,{error,emfile}},
                                  [{gen_server,terminate,6},
                                   {proc_lib,init_p_do_apply,3}]}},
                          {ancestors,
                              ['tcp_acceptor_sup_0.0.0.0:5672',<0.2387.0>,
                               rabbit_sup,<0.102.0>]},
                          {messages,[]},
                          {links,[<0.2388.0>]},
                          {dictionary,[]},
                          {trap_exit,false},
                          {status,running},
                          {heap_size,377},
                          {stack_size,24},
                          {reductions,597485}],
                         []]}}
** When handler state ==
{<0.48.0>,"/var/log/rabbitmq/rabbit-sasl.log",error}
** Reason == {'module could not be loaded',
                 [{lib,format_exception,
                      [5,exit,
                       {cannot_accept,{error,emfile}},

[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}],

#Fun<proc_lib.0.17830618>,#Fun<proc_lib.1.112399459>]},
                  {proc_lib,format_exception,3},
                  {proc_lib,format_rep,1},
                  {proc_lib,format_rep,1},
                  {proc_lib,format_rep,1},
                  {proc_lib,format,1},
                  {sasl_report,write_report2,4},
                  {sasl_report_file_h,handle_event,2}]}

=ERROR REPORT==== 16-Sep-2009::11:51:21 ===
Mnesia(rabbit at echonest03): ** ERROR ** (could not write core file: emfile)
 ** FATAL ** Cannot open log file
"/var/lib/rabbitmq/mnesia/rabbit/rabbit_durable_route.TMP": {file_error,


"/var/lib/rabbitmq/mnesia/rabbit/rabbit_durable_route.TMP",

                  emfile}

=INFO REPORT==== 16-Sep-2009::11:51:21 ===
    application: mnesia
    exited: shutdown
    type: temporary

=ERROR REPORT==== 16-Sep-2009::11:51:22 ===
** Generic server <0.1742.15> terminating
** Last message in was {'$gen_cast',
                           {method,
                               {'basic.get',1,

<<"blank_renderer_queue_priority">>,false},
                               none}}
** When Server state ==
{ch,running,1,<0.1727.15>,<0.1740.15>,undefined,none,
                            {set,0,16,16,8,80,48,
                                 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                  [],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]}}},
                            1,
                            {[],[]},
                            {[],[]},
                      ...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090916/ec8339bf/attachment.htm