[rabbitmq-discuss] RabbitMQ crash after few hours

[Artsiom]

Hello everybody.

I have RabbitMQ cluster across 4 machines. Then 2 machines were added to 
cluster.
Everything was ok, but after few hours of load three of them crashed 
almost simulteneously (according to log).
Besides at crash time memory consumption increased 3 times.

Log files from crashed machines:

(dell33) - machine 1
http://pastebin.com/A1rdBSkN

# rabbitmqctl cluster_status
Error: unable to connect to node wosnfs at dell33: nodedown
diagnostics:
- nodes and their ports on dell33: [{rabbitmqctl15567,47802},
                                     {rabbitmqctl15576,41488},
                                     {rabbitmqctl16776,44227},
                                     {rabbitmqctl16634,41524},
                                     {rabbitmqctl20944,36980},
                                     {rabbitmqctl20942,41667},
                                     {rabbitmqctl8262,44050}]
- current node: rabbitmqctl8262 at dell33
- current node home dir: /var/lib/rabbitmq
- current node cookie hash: FwHV/NurYGNpetHE+jQlMQ==

(dell34) - machine 2
http://pastebin.com/36ihKNqv

# rabbitmqctl list_queues

/usr/lib/rabbitmq/bin/rabbitmq-env: fork: retry: Resource temporarily 
unavailable
/usr/lib/rabbitmq/bin/rabbitmq-env: fork: retry: Resource temporarily 
unavailable
/usr/lib/rabbitmq/bin/rabbitmq-env: fork: retry: Resource temporarily 
unavailable
/usr/lib/rabbitmq/bin/rabbitmq-env: fork: retry: Resource temporarily 
unavailable

(dell38) - machine 3
http://pastebin.com/TfWBcSus

Log file from one of survived machines:

(dell39) - machine 4
http://pastebin.com/ij1k4L9K

Above mentioned issue appeared on rabbitmq 2.6.1, but sometimes it 
appears on 2.7.1.
Queues, which names are in log (queue-batch-{3,4,5,6}), are durable with 
"x-ha-policy" property.
All rabbitmq brokers are clustered as disk nodes.

Could anybody help me to examine log files and find out the reason?

--
Best regards,
Artsiom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120222/cca49afc/attachment.htm>