[rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection

[joseph rouphael]

Thanks Jason for sharing these articles. I will get a look on them to see
if they can help.


On Thu, Apr 17, 2014 at 7:54 PM, Jason McIntosh <mcintoshj at gmail.com> wrote:

> Watch also NUMA stuff on these bigger multi-core systems.  I've seen some
> very interesting effects due to NUMA and how it affects erlang.  Look at
> these posts for some more details:
>
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-January/024790.html
>
> Typically a system with as many cores as you've got requires some more
> "tweaking" to get good performance out of it.  One of the best articles I'd
> found was this:
> http://www.agoragames.com/blog/2011/06/24/of-penguins-rabbits-and-buses/
>
>
> I don't THINK this would have an impact on your issues with inter-node
> communications but I have to toss it out there.  Also, another fun thing we
> saw was long DNS lookups causing some odd issues as well.
> Jason
>
>
> On Thu, Apr 17, 2014 at 8:56 AM, Matthias Radestock <matthias at rabbitmq.com
> > wrote:
>
>> On 15/04/14 14:20, joseph rouphael wrote:
>>
>>> On Mon, Apr 14, 2014 at 12:00 PM, Matthias Radestock
>>> <matthias at rabbitmq.com <mailto:matthias at rabbitmq.com>> wrote:
>>>     What happens when you run the above test with both nodes on the same
>>>     machine?
>>>
>>> The result is almost the same: Fluctuating between 10 and 23KHZ
>>>
>>
>> Thanks for running all these tests. I've reproduced your observations.
>>
>> What we are seeing here is, at least partially, an issue with the Erlang
>> scheduler.
>>
>> I noticed that when running the test with PerfTest, single producer,
>> single consumer, two nodes, the consuming node was using 100% CPU, i.e. the
>> equivalent of a single core. That is usually an indication of an Erlang
>> scheduler issue since there is an appreciable degree of parallelism that
>> should be exploitable.
>>
>> For this particular test I managed to double performance by supplying the
>> "+swt very_low" or "+sfwi 1" flags to Erlang via RABBITMQ_SERVER_ERL_ARGS.
>>
>> So these flags, plus the other +s... flags of the Erlang VM, are worth
>> playing with.
>>
>>
>> Matthias.
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
>
> --
> Jason McIntosh
> https://github.com/jasonmcintosh/
> 573-424-7612
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140422/c3861aac/attachment.html>