[rabbitmq-discuss] Crash Report

[Matt Long]

Using RabbitMQ v2.8.5 on Ubuntu 11.10. Yes, I'm aware that version 3.0+ is 
now available...I'm just a bit behind in staying up to date :-)

RabbitMQ suddenly became unreachable by my clients. In particular, the amqp 
Python package raised ConnectionError: 541: (INTERNAL_ERROR, (0, 0), None). 
First thing I did was attempting to list the queues:

$ sudo rabbitmqctl list_queues
Listing queues ...
Error: {aborted,{no_exists,[rabbit_queue,
                            {amqqueue,{resource,<<"/">>,queue,'_'},
                                      '_','_','_','_','_','_','_'}]}}

This was fairly cryptic to me since I'm not too familiar with the innards 
of RabbitMQ. Next I checked the status:

$ sudo service rabbitmq-server status
Status of node 'rabbit at ip-10-100-105-118' ...
[{pid,879},
 {running_applications,[{rabbit,"RabbitMQ","2.8.5"},
                        {os_mon,"CPO  CXC 138 46","2.2.5"},
                        {sasl,"SASL  CXC 138 11","2.1.9.3"},
                        {stdlib,"ERTS  CXC 138 10","1.17.3"},
                        {kernel,"ERTS  CXC 138 10","2.14.3"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R14B02 (erts-5.8.3) [source] [64-bit] [smp:8:8] 
[rq:8] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,128970944},
          {processes,49035088},
          {processes_used,36245608},
          {system,79935856},
          {atom,1182377},
          {atom_used,1157674},
          {binary,17554600},
          {code,11329658},
          {ets,47162232}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,2920456192},
 {disk_free_limit,1000000000},
 {disk_free,6324633600},
 {file_descriptors,[{total_limit,924},
                    {total_used,186},
                    {sockets_limit,829},
                    {sockets_used,128}]},
 {processes,[{limit,1048576},{used,1235}]},
 {run_queue,0},
 {uptime,4829636}]
...done.

The error logs contain quite a bit of information related to the crash. 
Here are links to excerpts from the main RabbitMQ log and the sasl log:

rabbit at myhostname.log: http://pastebin.com/1CBjmbGb
rabbit at myhostname-sasl.log: http://pastebin.com/8qZas7hA

Stopping and then starting the rabbitmq-server daemon luckily fixed the 
problem and seemed to maintain at least most of Rabbit's state. Can someone 
help me interpret what happened and what I can do to prevent this in the 
future?

Many thanks in advance!

Cheers,
Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121217/acb12e38/attachment.htm>