One thing we found out is that disk-IO is the bottleneck in our setup (most queues are ACK-ed, HA- queues). Our solutions:<br>1) batching of messages improved performance (message size bigger, but less messages)<br>2) Faster disks (10k spinners RAIDed) improved performance<br>
<br>So make sure you also look at that part of the system.<br><br>Cheers<br>Maze<br><br><br><div class="gmail_quote">On Thu, Dec 20, 2012 at 6:27 PM, Chris Schmidt <span dir="ltr"><<a href="mailto:cischmidt77@gmail.com" target="_blank">cischmidt77@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<div class="im"><br>
On Dec 20, 2012, at 9:07 AM, Matthias Radestock <<a href="mailto:matthias@rabbitmq.com">matthias@rabbitmq.com</a>> wrote:<br>
<br>
> Chris,<br>
><br>
> On 20/12/12 16:18, Chris Schmidt wrote:<br>
>> There are 22 queues<br>
> > [...]<br>
>> A vendor let us borrow the 40-core machine to see if it would work<br>
>> better. �[...] Each core is only 11-15% utilized.<br>
><br>
> You may not be able to saturate 40 cores with 22 queues. Depends on how many producers and consumers there are.<br>
><br>
> The other constraint could be scheduling and general Erlang multi-core performance. This has improved *considerably* over the last few years. So please make sure you are running the latest Erlang/OTP release - R15B03.<br>
><br>
> You may also want to tweak the various smp/scheduling related settings in Erlang. In particular I have seen massive performance differences when adjusting the +swt setting.<br>
><br>
<br>
</div>RHEL 6.2 doesn't seem to have the latest version of Erlang available in the repositories. Does anyone have experience with manually building Erlang on that platform with success? We're on R14B04 right now. There are a huge number of erlang related rpms that get installed I'm not certain if a single Erlang version for CentOS would work properly or not.<br>
<br>
I had +K true -smp enable +native +sbtps plus the other RMQ defaults set on the 40 core machine while testing. I also removed everything to use plain vanilla RMQ settings to see how that impacted performance.<br>
<br>
I also had 15 consumers running against the initial queue that was having problems. I also tried sending the data into 2 - 40 exchanges (and updated the queue to read from those exchanges) in order to see if I could spread the load that way. Does a single process manage each queue? If so I can see that being a potential bottleneck due to the core's speed difference.<br>
<br>
Is there an erlang or rabbitmqctl command that I can run against the RMQ server to determine if the queues themselves are the bottleneck?<br>
<div class="im"><br>
>> I've tried HiPE in the past, but the system % jumped considerably<br>
>> and we actually got poorer performance and instability.<br>
><br>
> That's odd. Again, please make sure you are runninng the latest Erlang; HiPE is another area that has improved considerably recently.<br>
<br>
</div>I'd definitely like to try HiPE again since we're CPU bound on the existing production server.<br>
<div class="im">><br>
>> I know the cores are slower in overall speed, but I would have bet<br>
>> money that in �aggregate this machine would outperform our other one.<br>
><br>
> Depends on the workload. e.g. if queues are the bottleneck then having many more (slow) cores than queues won't help.<br>
><br>
> Regards,<br>
><br>
> Matthias.<br>
<br>
</div>I appreciate the pointers, thanks Matthias!<br>
<div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br>