[rabbitmq-discuss] RMQ performance between high MHz/low core vs lower MHz/high core servers

[Chris Schmidt]

I've been given a chance to try a high core server (4x10) as a possible RMQ
server and have hit a few throughput issues that I hope someone can  help
shed some light on.

We currently have a decently powered single RMQ server that is running as a
standalone node and has RHEL 6.2. It has 12 cores running at 3.4GHz. There
are 22 queues, some configured with no-ack, but mostly with acks being
used. The message sizes are varied but can get large and we hit about 51k
messages/sec throughput through all queues. During the day this system caps
out at 63% CPU utilization (13% system) and stays that way until the
evening. We are definitely hitting a bottleneck with it. I've tried HiPE in
the past, but the system % jumped considerably and we actually got poorer
performance and instability.

A vendor let us borrow the 40-core machine to see if it would work better.
When I switched over to it using the same configuration as the production
machine, it capped out at around 12,000 messages/sec. We changed the
network card and tweaked MTU to see if that was causing a problem, but it
still seems capped. What's odd is I can't determine what is causing the
bottleneck on the machine. Each core is only 11-15% utilized. There is no
iowait at all. The machine does have a lot more RAM at 256 GB vs 72 GB on
the existing hardware. We disabled NUMA in the bios to see if that would
change things but we got the same performance. On the producer side that
injects the initial data, the logs show it can't push data into the new
machine fast enough. There are in-memory queues that store data in case of
slow downs and they immediately begin to fill up when pushing to the new
machine, whereas it rarely happens with the old one.

Are there any other tools that I can use to see why the erlang processes
don't seem to be fully utilizing the server's capacity? I know the cores
are slower in overall speed, but I would have bet money that in aggregate
this machine would outperform our other one.

Thanks in advance for any suggestions,

 Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121220/8dc87930/attachment.htm>