[rabbitmq-discuss] Debugging rabbitmq crash

[Matt Calder]

Matthew,

I had not checked, but, the client was writing right up to 6:27. The
client was processing a very large job at the time of the memory
warning, so, that makes sense. But it appears to have pushed on and
processed several hundred more jobs (each job results in 10-100+
message exchanges between client and server).

Matt



On Fri, Jun 25, 2010 at 12:16 PM, Matthew Sackman <matthew at rabbitmq.com> wrote:
> Hi Matt,
>
> On Fri, Jun 25, 2010 at 12:00:07PM -0400, Matt Calder wrote:
>> =INFO REPORT==== 23-Jun-2010::01:59:00 ===
>>    alarm_handler: {set,{system_memory_high_watermark,[]}}
>>
>> =INFO REPORT==== 23-Jun-2010::02:02:55 ===
>>    alarm_handler: {clear,system_memory_high_watermark}
>>
>> =WARNING REPORT==== 23-Jun-2010::06:27:37 ===
>> exception on TCP connection <0.15425.0> from 127.0.0.1:47025
>> connection_closed_abruptly
>>
>> =INFO REPORT==== 23-Jun-2010::06:27:37 ===
>> closing TCP connection <0.15425.0> from 127.0.0.1:47025
>
> This is very interesting. Rabbit did run out of memory, but then later
> recovered. And then some 4 hours later, the client was kicked off. The
> python client doesn't handle channel.flow (the server raises this when
> it runs out of memory which is asking the client to stop publishing
> messages - see http://www.rabbitmq.com/extensions.html#memsup). Do you
> have any evidence that the client managed to do anything at all between
> 2am and 6:30am? I wonder if there is some very long timeout somewhere
> and that you're actually hitting a limitation in the python client which
> for some reason takes 4.5 hours until the client gets kicked off. Lots
> of guessing here though...
>
> Matthew
>