[rabbitmq-discuss] Taking error_logger out of the "fast path" (was Re: beam process using 100% cpu and lots of memory)
majek04 at gmail.com
Thu Feb 25 16:49:47 GMT 2010
On Thu, Feb 25, 2010 at 08:42, Tony Garnock-Jones <tonyg at lshift.net> wrote:
> Matthew Sackman wrote:
>> That's potentially a problem. We've recently discovered that the error
>> logger, as it logs a couple of messages for every connection, isn't very
>> fast, and its mailbox can rapidly grow.
> That's very interesting. Should we avoid it? It's a tradeoff between the
> nice features, flexibility & integration using the OTP error logging
> mechanism gives us and the overhead of same. I'm sure we could gin up a
> sane logger that's much quicker (but less flexible) fairly quickly.
> What if, instead, we used syslog? Or simply logged everything only
> through an internal exchange, avoiding the error_logger?
I think we can take several approaches:
- use syslog and/or rewrite the logging infrastructure
- invent filter on top of the logger, which would stop sending
new errors if there are too many of them. Something similar
to "last message repeated XX time(s)" from syslog.
- move logger away from the critical path, by doing logging
asynchronously. This can lead to out-of-memory crash.
More information about the rabbitmq-discuss