[rabbitmq-discuss] Debugging rabbitmq crash

[Matthew Sackman]

Hi Matt,

On Fri, Jun 25, 2010 at 12:00:07PM -0400, Matt Calder wrote:
> =INFO REPORT==== 23-Jun-2010::01:59:00 ===
>    alarm_handler: {set,{system_memory_high_watermark,[]}}
> 
> =INFO REPORT==== 23-Jun-2010::02:02:55 ===
>    alarm_handler: {clear,system_memory_high_watermark}
> 
> =WARNING REPORT==== 23-Jun-2010::06:27:37 ===
> exception on TCP connection <0.15425.0> from 127.0.0.1:47025
> connection_closed_abruptly
> 
> =INFO REPORT==== 23-Jun-2010::06:27:37 ===
> closing TCP connection <0.15425.0> from 127.0.0.1:47025

This is very interesting. Rabbit did run out of memory, but then later
recovered. And then some 4 hours later, the client was kicked off. The
python client doesn't handle channel.flow (the server raises this when
it runs out of memory which is asking the client to stop publishing
messages - see http://www.rabbitmq.com/extensions.html#memsup). Do you
have any evidence that the client managed to do anything at all between
2am and 6:30am? I wonder if there is some very long timeout somewhere
and that you're actually hitting a limitation in the python client which
for some reason takes 4.5 hours until the client gets kicked off. Lots
of guessing here though...

Matthew