[rabbitmq-discuss] Performance tests and a warning on HIPE

[Tim Watson]

Hi Jason,

On 11 Mar 2013, at 14:16, Jason McIntosh wrote:
> I'd love to be able to provide you more details but I've not gotten a crash dump yet on any of the systems I've seen this (that's 3 in total, one VM, two different physical boxes).  Any suggestions on how to get some more data out of erlang/rabbit for this?
> 

As you're not getting crash dump, I'd suggest going for a core dump of the emulator instead. First you'll want to try (not as root) to change the max core dump file size to unlimited

$ ulimit -c unlimited

That'll likely fail on permissions, then check, that you don't set ulimit -c 0 in any shell configuration files for the account under which the broker will run - this should be the rabbitmq user.

Globally enabling core dumps must be done as root, usually in /etc/security/limits.conf, e.g.,

# /etc/security/limits.conf
#
# Each line describes a limit for a user in the form:
#
# <domain> <type> <item> <value>
#
*  soft  core  unlimited

Note that there are other places you might need to look such as /etc/profile (on RHEL/OEL)

# No core files by default
# ulimit -S -c 0 > /dev/null 2>&1

... and /etc/init.d/functions (RHEL/OEL)

# make sure it doesn't core dump anywhere unless requested
# ulimit -S -c ${DAEMON_COREFILE_LIMIT:-0} >/dev/null 2>&1

And so on. You can check it has worked like so:

$ ulimit -a

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
[etc...]

To read the core properly, it might be necessary to get the binaries as well (i.e., beam.smp etc). Depending on how you've installed Erlang, that could be interesting, but we can cross that bridge when we come to it.

More likely than not, we'd be keen to get that whole core dump over to the OTP team, so in the first instance once if/when you've got a core dump just get in touch and we'll head over to the erlang-bugs list together and engage them from there. Bare in mind that a core dump may contain all manner of sensitive information. The OTP team will usually sign a relevant legal document to protect your rights if that's necessary.

Cheers,
Tim