[rabbitmq-discuss] Taking error_logger out of the "fast path" (was Re: beam process using 100% cpu and lots of memory)

[majek04]

On Thu, Feb 25, 2010 at 08:42, Tony Garnock-Jones <tonyg at lshift.net> wrote:
> Matthew Sackman wrote:
>> That's potentially a problem. We've recently discovered that the error
>> logger, as it logs a couple of messages for every connection, isn't very
>> fast, and its mailbox can rapidly grow.
>
> That's very interesting. Should we avoid it? It's a tradeoff between the
> nice features, flexibility & integration using the OTP error logging
> mechanism gives us and the overhead of same. I'm sure we could gin up a
> sane logger that's much quicker (but less flexible) fairly quickly.
>
> What if, instead, we used syslog? Or simply logged everything only
> through an internal exchange, avoiding the error_logger?

I think we can take several approaches:
 - use syslog and/or rewrite the logging infrastructure
 - invent filter on top of the logger, which would stop sending
   new errors if there are too many of them. Something similar
   to "last message repeated XX time(s)" from syslog.
 - move logger away from the critical path, by doing logging
   asynchronously. This can lead to out-of-memory crash.

Marek