[rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection

[joseph rouphael]

On Mon, Apr 14, 2014 at 12:00 PM, Matthias Radestock
<matthias at rabbitmq.com>wrote:

> On 11/04/14 17:37, joseph rouphael wrote:
>
>> Test 8: Dual node, Queues/Consumers on same node, 20 queue, ACK off
>> -------------------------------------------------------------------
>> Nodes: Double node
>> Queues created on: Node1
>> Producers on: node 2
>> Consumers on: node 1
>> Producer Processes: 20
>> Producer regulate rate per process: 5KHZ
>> Acknowledgment: OFF
>> Expected rate: 100KHZ
>> Achieved rate: Fluctuating between 10KHZ and 20KHZ
>> CPU idle: 95%
>> Note: Bottleneck on inter-node connection
>>
>
> What happens when you run the above test with both nodes on the same
> machine?
>

The result is almost the same: Fluctuating between 10 and 23KHZ (Expected
100KHZ). Here is below the summary:

Test 11: Dual nodes same machine, Queues/Consumers on same node, 20 queue,
ACK on/off
------------------------------------------------------------
-------------------------------------------------------------------------------
Nodes: Double nodes same machine
Queues created on: Node1
Producers on: node 2
Consumers on: node 1
Producer Processes: 20
Producer regulate rate per process: 5KHZ
Acknowledgment: OFF or ON same result
Expected rate: 100KHZ
Achieved rate: Fluctuating between 10KHZ and 23KHZ
CPU idle: 82%
Note: Bottleneck on inter-node connection



> Also, you really shouldn't need 20 producers/consumers to saturate a
> broker, unless the machine has >20 cores. So I'd be interested to hear what
> the smallest producer/consumer count is at which you see a significant
> performance difference between the single node and dual node case.
>

The machine I am working on has 32 cores. So I am relaxed with the numbers
of processes.
1 producer/consumer achieve a maximum of 11 KHZ (No inter-node impact. On
single node it achieves the same)
2 producers/consumers achieve a maximum of ~ 20 KHZ (Minor inter-node
impact - expected 22KHZ)
3 producers/consumers achieve a maximum of ~ 20 KHZ (Major inter-node
impact - expected 33KHZ)

>
> Finally, it would be helpful if you ran your tests using RabbitMQ's
> standard performance testing tool, PerfTest -
> http://www.rabbitmq.com/java-tools.html#perftest. In order to replicate
> the above scenario you'd run 20 PerfTest instances with options
>   -u q<n> -x 0 -a
> and 20 instances with
>   -u q<n> -y 0
> with n in 1..20.
>
>
perftest ran as advised and the test produced the same result. There is a
real inter-node bottleneck.

Results of perftest:

Test 1: Double nodes same machine
----------
Node1 and Node2 on same machine
Node1 listening on port 5672
Node2 listening on port 5673
20 consumers running full speed
20 producers running full speed
Message size 1KB

Max rate achieved: 16KHZ
CPU Idle 92%
Note: Inter-node bottleneck

The script used to run the test:

#!/bin/sh

for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
do
./runjava.sh com.rabbitmq.examples.PerfTest -k queue$i -x 0 -a -h amqp://
193.100.200.130:5673 -s 1000 &
./runjava.sh com.rabbitmq.examples.PerfTest -k queue$i -y 0 -h amqp://
193.100.200.130:5672 -s 1000 &
done

Test 2: Single node
---------
Single node: Node1
Node1 listening on port 5672
20 consumers running full speed
20 producers running full speed
Message size 1KB

Max rate achieved: 200KHZ
CPU Idle 12%

The script used to run the test:

#!/bin/sh

for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
do
./runjava.sh com.rabbitmq.examples.PerfTest -k queue$i -x 0 -a -h amqp://
193.100.200.130:5672 -s 1000 &
./runjava.sh com.rabbitmq.examples.PerfTest -k queue$i -y 0 -h amqp://
193.100.200.130:5672 -s 1000 &
done



>
> Regards,
>
> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140415/2277a5b6/attachment.html>