[rabbitmq-discuss] rabbitmq-server halting with rabbit_sasl_report_file_h crash

[Patrick Galvin]

Hi,

I have been testing rabbitmq for a potential use at work. I am firing about
500 messages/sec (estimate) from 60 or so processes.
All starts well (I can see all the messages being pumped out, all going
quickly) and after a few minutes rabbitmq halts with the error below - '**
gen_event handler rabbit_sasl_report_file_h crashed. '
(same error keeps occuring).

Infrastructure wise I have a simple queue setup with one consumer (erlang
client)  with the producer a java client; sending very short messages
The box is running a single rabbitmq server process and is a quad core, 2
gig box

Things I have tried so far:
Increase the ulimit to 100,000+
Increase the number of consumers to 3 to try and take any load off the
rabbitmq server
Use ERL_MAX_PORTS 10240 / ERL_MAX_ETS_TABLES 10240

Any help would be great,

thanks

Patrick

=ERROR REPORT==== 5-Aug-2009::01:00:42 ===
** Generic server <0.141.0> terminating
** Last message in was {inet_async,#Port<0.1999>,16213,{ok,#Port<0.5041>}}
** When Server state == none
** Reason for termination ==
** {cannot_accept,{error,emfile}}

=INFO REPORT==== 5-Aug-2009::01:00:42 ===
starting TCP connection <0.4894.0> from 192.168.36.72:33078

=INFO REPORT==== 5-Aug-2009::01:00:42 ===
starting TCP connection <0.4895.0> from 192.168.36.72:33079

=ERROR REPORT==== 5-Aug-2009::01:00:42 ===
File operation error: emfile. Target: ./lib.beam. Function: get_file.
Process: code_server.

=ERROR REPORT==== 5-Aug-2009::01:00:42 ===
File operation error: emfile. Target:
/usr/lib/erlang/lib/kernel-2.13.2/ebin/lib.beam. Function: get_file.
Process: code_server.

=ERROR REPORT==== 5-Aug-2009::01:00:42 ===
File operation error: emfile. Target:
/usr/lib/erlang/lib/stdlib-1.16.2/ebin/lib.beam. Function: get_file.
Process: code_server.

.... (every lib directory in my install)

=ERROR REPORT==== 5-Aug-2009::01:00:42 ===
File operation error: emfile. Target:
/usr/lib/erlang/lib/appmon-2.1.10.1/ebin/lib.beam. Function: get_file.
Process: code_server.

=ERROR REPORT==== 5-Aug-2009::01:00:42 ===
File operation error: emfile. Target: lib.beam. Function: get_file. Process:
code_server.

=ERROR REPORT==== 5-Aug-2009::01:00:42 ===
** gen_event handler rabbit_sasl_report_file_h crashed.
** Was installed in error_logger
** Last event was: {error_report,<0.102.0>,
                       {<0.141.0>,crash_report,

[[{initial_call,{tcp_acceptor,init,['Argument__1']}},
                          {pid,<0.141.0>},
                          {registered_name,[]},
                          {error_info,
                              {exit,
                                  {cannot_accept,{error,emfile}},
                                  [{gen_server,terminate,6},
                                   {proc_lib,init_p_do_apply,3}]}},
                          {ancestors,
                              ['tcp_acceptor_sup_0.0.0.0:5672',<0.138.0>,
                               rabbit_sup,<0.103.0>]},
                          {messages,[]},
                          {links,[<0.139.0>]},
                          {dictionary,[]},
                          {trap_exit,false},
                          {status,running},
                          {heap_size,987},
                          {stack_size,24},
                          {reductions,147211}],
                         []]}}
** When handler state ==
{<0.49.0>,"/var/log/rabbitmq/rabbit-sasl.log",error}
** Reason == {'module could not be loaded',
                 [{lib,format_exception,
                      [5,exit,
                       {cannot_accept,{error,emfile}},

[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}],

#Fun<proc_lib.0.17830618>,#Fun<proc_lib.1.112399459>]},
                  {proc_lib,format_exception,3},
                  {proc_lib,format_rep,1},
                  {proc_lib,format_rep,1},
                  {proc_lib,format_rep,1},
                  {proc_lib,format,1},
                  {sasl_report,write_report2,4},
                  {sasl_report_file_h,handle_event,2}]}

=WARNING REPORT==== 5-Aug-2009::01:03:19 ===
exception on TCP connection <0.736.0> from 192.168.160.203:62435
connection_closed_abruptly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090805/be4ea11e/attachment.htm