[rabbitmq-discuss] Server hanging on start, randomly dying

[Jason McIntosh]

So first, my environment:
RabbitMQ 3.0.4
Oracle linux 6.2
Cluster of 3 nodes, 2 disc, 1 ram.

The problem I'm hitting is I've got one of my nodes hanging and dying.  I
got it to start and about a minute later it dies again.  I have no reports
in any log files, and not a clue what the heck is going on.

At first, I thought it might be tied to some invalid commands, i.e.
rabbitmqctl list_status -p SomeNonExistentQueue (which seems to have
problems regardless).  That command though works on one of the other nodes.

Any advice here?  Nothing as state in the logs.  I've hung in various
places, for example:
-- rabbit boot start
starting file handle cache server
...done
starting worker pool
 ...done
starting database                                                     ...

I get here and things stop.  Other times the whole server starts but then
nothing is working.  This cluster has been in a good state for weeks and
all of a sudden has started doing this.  In non detached mode, by starting
rabbitmq-server AS root by hand, I have the process running.  Below is the
results after a rabbitmqctl status that I managed to get before the system
crashed, as well as the list of plugins.  The last thing in the log file is
"Management plugin started.  Port: 15672"

Thanks for any advice!
Jason

{erlang_version,
     "Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:24:24]
[async-threads:30] [hipe] [kernel-poll:true]\n"},
 {memory,
     [{total,39045064},
      {connection_procs,5664},
      {queue_procs,792848},
      {plugins,163256},
      {other_proc,11498766},
      {mnesia,404168},
      {mgmt_db,18304},
      {msg_index,94904},
      {other_ets,1224192},
      {binary,42552},
      {code,18246956},
      {atom,703377},
      {other_system,5850077}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,54118517964},
 {disk_free_limit,1000000000},
 {disk_free,840619569152},
 {file_descriptors,

 [{total_limit,924},{total_used,8},{sockets_limit,829},{sockets_used,1}]},
 {processes,[{limit,1048576},{used,316}]},
 {run_queue,0},
 {uptime,14}]


HEre's a list of installed plugins:
[e] amqp_client                       3.0.4
[ ] cowboy                            0.5.0-rmq3.0.4-git4b93c2d
[ ] eldap                             3.0.4-gite309de4
[e] mochiweb                          2.3.1-rmq3.0.4-gitd541e9a
[ ] rabbitmq_auth_backend_ldap        3.0.4
[ ] rabbitmq_auth_mechanism_ssl       3.0.4
[ ] rabbitmq_consistent_hash_exchange 3.0.4
[E] rabbitmq_federation               3.0.4
[E] rabbitmq_federation_management    3.0.4
[ ] rabbitmq_jsonrpc                  3.0.4
[ ] rabbitmq_jsonrpc_channel          3.0.4
[ ] rabbitmq_jsonrpc_channel_examples 3.0.4
[E] rabbitmq_management               3.0.4
[E] rabbitmq_management_agent         3.0.4
[E] rabbitmq_management_visualiser    3.0.4
[ ] rabbitmq_mqtt                     3.0.4
[ ] rabbitmq_old_federation           3.0.4
[E] rabbitmq_shovel                   3.0.4
[E] rabbitmq_shovel_management        3.0.4
[ ] rabbitmq_stomp                    3.0.4
[ ] rabbitmq_tracing                  3.0.4
[e] rabbitmq_web_dispatch             3.0.4
[ ] rabbitmq_web_stomp                3.0.4
[ ] rabbitmq_web_stomp_examples       3.0.4
[ ] rfc4627_jsonrpc                   3.0.4-git7ab174b
[ ] sockjs                            0.3.4-rmq3.0.4-git3132eb9
[e] webmachine                        1.9.1-rmq3.0.4-git52e62bc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130516/061906da/attachment.htm>