[rabbitmq-discuss] Debugging rabbitmq crash

Wed Jun 23 11:48:25 BST 2010

I am looking for guidance on how to debug a crash of rabbit. The
situation is this. There is a pair of processes running on a single
machine. One process sends a message to the other, and the other
responds. This is happening at ~20 message pairs per second. There is
a single direct exchange and two queues, one for messages going one
way, the other for messages going the other way. The queues are not
durable, and not auto delete. The processes run for ~15 hours without
issue over a single connection. Then the connection closes. Here is
the rabbit.log:

 =INFO REPORT==== 22-Jun-2010::18:03:03 ===
accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:55824

=INFO REPORT==== 22-Jun-2010::18:03:03 ===
starting TCP connection <0.15621.0> from 127.0.0.1:55824

=INFO REPORT==== 23-Jun-2010::01:59:00 ===
    alarm_handler: {set,{system_memory_high_watermark,[]}}

=INFO REPORT==== 23-Jun-2010::02:02:55 ===
    alarm_handler: {clear,system_memory_high_watermark}

=WARNING REPORT==== 23-Jun-2010::06:27:37 ===
exception on TCP connection <0.15425.0> from 127.0.0.1:47025
connection_closed_abruptly

=INFO REPORT==== 23-Jun-2010::06:27:37 ===
closing TCP connection <0.15425.0> from 127.0.0.1:47025

=WARNING REPORT==== 23-Jun-2010::06:27:37 ===
exception on TCP connection <0.15621.0> from 127.0.0.1:55824
connection_closed_abruptly

=INFO REPORT==== 23-Jun-2010::06:27:37 ===
closing TCP connection <0.15621.0> from 127.0.0.1:55824

So, there appeared to be a memory issue, but it also appeared to pass.
I watched the process on and off and it seemed to be using a steady
15% of available memory according to top. All the other logs are
either old, or empty. Specifically, rabbit-sasl.log was touched
apparently at the time of the crash.

-rw-r--r-- 1 rabbitmq rabbitmq     0 2010-06-23 06:27 rabbit-sasl.log

I can reproduce this, though it takes the 15 hours or so.

If anyone can guide me through the steps necessary to debugging this I
would appreciate it. Specific suggestions, like, "your problem is X",
are of course welcome, but I am also interested in process. Thank you,

Matt