[rabbitmq-discuss] RMQ performance between high MHz/low core vs lower MHz/high core servers

[Matthias Reik]

One thing we found out is that disk-IO is the bottleneck in our setup (most
queues are ACK-ed, HA- queues). Our solutions:
1) batching of messages improved performance (message size bigger, but less
messages)
2) Faster disks (10k spinners RAIDed) improved performance

So make sure you also look at that part of the system.

Cheers
Maze


On Thu, Dec 20, 2012 at 6:27 PM, Chris Schmidt <cischmidt77 at gmail.com>wrote:

> Hello,
>
> On Dec 20, 2012, at 9:07 AM, Matthias Radestock <matthias at rabbitmq.com>
> wrote:
>
> > Chris,
> >
> > On 20/12/12 16:18, Chris Schmidt wrote:
> >> There are 22 queues
> > > [...]
> >> A vendor let us borrow the 40-core machine to see if it would work
> >> better.  [...] Each core is only 11-15% utilized.
> >
> > You may not be able to saturate 40 cores with 22 queues. Depends on how
> many producers and consumers there are.
> >
> > The other constraint could be scheduling and general Erlang multi-core
> performance. This has improved *considerably* over the last few years. So
> please make sure you are running the latest Erlang/OTP release - R15B03.
> >
> > You may also want to tweak the various smp/scheduling related settings
> in Erlang. In particular I have seen massive performance differences when
> adjusting the +swt setting.
> >
>
> RHEL 6.2 doesn't seem to have the latest version of Erlang available in
> the repositories. Does anyone have experience with manually building Erlang
> on that platform with success? We're on R14B04 right now. There are a huge
> number of erlang related rpms that get installed I'm not certain if a
> single Erlang version for CentOS would work properly or not.
>
> I had +K true -smp enable +native +sbtps plus the other RMQ defaults set
> on the 40 core machine while testing. I also removed everything to use
> plain vanilla RMQ settings to see how that impacted performance.
>
> I also had 15 consumers running against the initial queue that was having
> problems. I also tried sending the data into 2 - 40 exchanges (and updated
> the queue to read from those exchanges) in order to see if I could spread
> the load that way. Does a single process manage each queue? If so I can see
> that being a potential bottleneck due to the core's speed difference.
>
> Is there an erlang or rabbitmqctl command that I can run against the RMQ
> server to determine if the queues themselves are the bottleneck?
>
> >> I've tried HiPE in the past, but the system % jumped considerably
> >> and we actually got poorer performance and instability.
> >
> > That's odd. Again, please make sure you are runninng the latest Erlang;
> HiPE is another area that has improved considerably recently.
>
> I'd definitely like to try HiPE again since we're CPU bound on the
> existing production server.
> >
> >> I know the cores are slower in overall speed, but I would have bet
> >> money that in  aggregate this machine would outperform our other one.
> >
> > Depends on the workload. e.g. if queues are the bottleneck then having
> many more (slow) cores than queues won't help.
> >
> > Regards,
> >
> > Matthias.
>
> I appreciate the pointers, thanks Matthias!
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121221/d7a4338e/attachment.htm>