[rabbitmq-discuss] RabbitMQ internal error

[Michael Arnoldus]

Hi Tony,

Yes, we're running on a Mac. Don't know about the CPU as I  
unfortunately wasn't present when I happened. I'll check to and see if  
somebody else knows. Good to hear you have changed the startup script  
and we'll be careful to save the log-file if it happens again and  
check CPU utilization and if we can get an erlang shell.

Thank you for taking this so seriously. I will send you anything we  
discover, and please let me know if you find anything that you think  
might be relevant to this, or any experiments you would like us to try.

Regards,

Michael


On Feb 28, 2008, at 8:52 , Tony Garnock-Jones wrote:

> Hi Michael,
>
> Michael Arnoldus wrote:
>> Yesterday we experienced another problem with RabbitMQ. Possibly  
>> still our own fault, but this time a bit more severe. Suddenly from  
>> out of the blue it was impossible to send a single message through  
>> Rabbit. Even restart of the components connecting to rabbit didn't  
>> help. The erlang process stayed but didn't seem to work. Killing  
>> the beam process helped and everything returned to normal.
>
> This is extremely interesting.
>
> - What architecture are you running on? Is it a Mac?
> - Was the CPU pinned to 100%?
> - Were you able to issue commands at the Erlang prompt in the server?
>
> We are tracking down what we suspect to be a Mac-specific bug in the  
> Erlang runtime that manifests in some corner-cases of socket  
> shutdown - it would be interesting if you have detected the same  
> thing we're chasing. (We are still in the early stages of our  
> investigation - we can't say for sure yet whether it's really a  
> runtime problem.)
>
>> In a log file we had:
>> ERROR    2008-02-26 16:17:32,857   --call got Closed:  
>> Method(name=close, id=60) (541, 'INTERNAL_ERROR', 0, 0) content =  
>> None
>
> If only the other log files hadn't been stomped on by the broker  
> startup! Your message has prompted us to fix this bad behaviour - we  
> have changed the startup scripts to move existing log files out of  
> the way, keeping the most recent few files.
>
> The INTERNAL_ERROR message is very interesting, because it indicates  
> a real bug in the broker. We don't see it in the case of the Mac bug  
> I mentioned earlier, so you might have found something different.
>
> This is probably the code that ran:
>
> lookup_amqp_exception(Other) ->
>    rabbit_log:warning("Non-AMQP exit reason '~p'~n", [Other]),
>    {true, ?INTERNAL_ERROR, <<"INTERNAL_ERROR">>, none}.
>
> ... which produces a "Non-AMQP exit reason" message in the log. I'm  
> afraid without that message, we'll have a tough time diagnosing this  
> one.
>
> Regards,
>  Tony