[rabbitmq-discuss] RabbitMQ internal error

[Tony Garnock-Jones]

Hi Michael,

Michael Arnoldus wrote:
> Yesterday we experienced another problem with RabbitMQ. Possibly still 
> our own fault, but this time a bit more severe. Suddenly from out of the 
> blue it was impossible to send a single message through Rabbit. Even 
> restart of the components connecting to rabbit didn't help. The erlang 
> process stayed but didn't seem to work. Killing the beam process helped 
> and everything returned to normal.

This is extremely interesting.

  - What architecture are you running on? Is it a Mac?
  - Was the CPU pinned to 100%?
  - Were you able to issue commands at the Erlang prompt in the server?

We are tracking down what we suspect to be a Mac-specific bug in the 
Erlang runtime that manifests in some corner-cases of socket shutdown - 
it would be interesting if you have detected the same thing we're 
chasing. (We are still in the early stages of our investigation - we 
can't say for sure yet whether it's really a runtime problem.)

> In a log file we had:
> ERROR    2008-02-26 16:17:32,857   --call got Closed: Method(name=close, 
> id=60) (541, 'INTERNAL_ERROR', 0, 0) content = None

If only the other log files hadn't been stomped on by the broker 
startup! Your message has prompted us to fix this bad behaviour - we 
have changed the startup scripts to move existing log files out of the 
way, keeping the most recent few files.

The INTERNAL_ERROR message is very interesting, because it indicates a 
real bug in the broker. We don't see it in the case of the Mac bug I 
mentioned earlier, so you might have found something different.

This is probably the code that ran:

lookup_amqp_exception(Other) ->
     rabbit_log:warning("Non-AMQP exit reason '~p'~n", [Other]),
     {true, ?INTERNAL_ERROR, <<"INTERNAL_ERROR">>, none}.

... which produces a "Non-AMQP exit reason" message in the log. I'm 
afraid without that message, we'll have a tough time diagnosing this one.

Regards,
   Tony