We have a series of machines that we boot at once that all consume and produce messages on about 20 different queues all to the same broker. The broker has been terminating when these machines boot recently. We're not sure why. Here's some log data:<div>
<br></div><div><div>=ERROR REPORT==== 16-Sep-2009::11:48:52 ===</div><div>** Generic server <0.2390.0> terminating </div><div>** Last message in was {inet_async,#Port<0.3893>,173,{ok,#Port<0.430698>}}</div>
<div>** When Server state == none</div><div>** Reason for termination == </div><div>** {cannot_accept,{error,emfile}}</div><div><br></div><div><br></div><div>=ERROR REPORT==== 16-Sep-2009::11:48:52 ===</div><div>** gen_event handler rabbit_sasl_report_file_h crashed.</div>
<div>** Was installed in error_logger</div><div>** Last event was: {error_report,<0.101.0>,</div><div> {<0.2390.0>,crash_report,</div><div> [[{initial_call,{tcp_acceptor,init,['Argument__1']}},</div>
<div> {pid,<0.2390.0>},</div><div> {registered_name,[]},</div><div> {error_info,</div><div> {exit,</div><div> {cannot_accept,{error,emfile}},</div>
<div> [{gen_server,terminate,6},</div><div> {proc_lib,init_p_do_apply,3}]}},</div><div> {ancestors,</div><div> ['tcp_acceptor_sup_0.0.0.0:5672',<0.2387.0>,</div>
<div> rabbit_sup,<0.102.0>]},</div><div> {messages,[]},</div><div> {links,[<0.2388.0>]},</div><div> {dictionary,[]},</div>
<div> {trap_exit,false},</div><div> {status,running},</div><div> {heap_size,377},</div><div> {stack_size,24},</div><div> {reductions,597485}],</div>
<div> []]}}</div><div>** When handler state == {<0.48.0>,"/var/log/rabbitmq/rabbit-sasl.log",error}</div><div>** Reason == {'module could not be loaded',</div><div> [{lib,format_exception,</div>
<div> [5,exit,</div><div> {cannot_accept,{error,emfile}},</div><div> [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}],</div><div> #Fun<proc_lib.0.17830618>,#Fun<proc_lib.1.112399459>]},</div>
<div> {proc_lib,format_exception,3},</div><div> {proc_lib,format_rep,1},</div><div> {proc_lib,format_rep,1},</div><div> {proc_lib,format_rep,1},</div><div>
{proc_lib,format,1},</div><div> {sasl_report,write_report2,4},</div><div> {sasl_report_file_h,handle_event,2}]}</div><div><br></div><div>=ERROR REPORT==== 16-Sep-2009::11:51:21 ===</div>
<div>Mnesia(rabbit@echonest03): ** ERROR ** (could not write core file: emfile)</div><div> ** FATAL ** Cannot open log file "/var/lib/rabbitmq/mnesia/rabbit/rabbit_durable_route.TMP": {file_error,</div><div> "/var/lib/rabbitmq/mnesia/rabbit/rabbit_durable_route.TMP",</div>
<div> emfile}</div><div><br></div><div>=INFO REPORT==== 16-Sep-2009::11:51:21 ===</div><div> application: mnesia</div><div> exited: shutdown</div>
<div> type: temporary</div><div><br></div><div>=ERROR REPORT==== 16-Sep-2009::11:51:22 ===</div><div>** Generic server <0.1742.15> terminating </div><div>** Last message in was {'$gen_cast',</div><div> {method,</div>
<div> {'basic.get',1,</div><div> <<"blank_renderer_queue_priority">>,false},</div><div> none}}</div>
<div>** When Server state == {ch,running,1,<0.1727.15>,<0.1740.15>,undefined,none,</div><div> {set,0,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> [],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div> [],[]}}},</div><div> 1,</div>
<div> {[],[]},</div><div> {[],[]},</div><div> ...</div><div><br></div></div>