[rabbitmq-discuss] Fwd: Performance bottleneck on inter-node connection

[joseph rouphael]

---------- Forwarded message ----------
From: joseph rouphael <josephrouphael at gmail.com>
Date: Fri, Apr 11, 2014 at 4:11 PM
Subject: Re: [rabbitmq-discuss] Performance bottleneck on inter-node
connection
To: Matthias Radestock <matthias at rabbitmq.com>





On Thu, Apr 10, 2014 at 7:01 PM, Matthias Radestock
<matthias at rabbitmq.com>wrote:

> (re-adding the list)
>
>
> On 10/04/14 16:42, joseph rouphael wrote:
>
>> On Thu, Apr 10, 2014 at 11:57 AM, Matthias Radestock
>> <matthias at rabbitmq.com <mailto:matthias at rabbitmq.com>> wrote:
>>     Have you tried this w/o hipe?
>> In fact, HIPE is not enabled in Erlang, so this config has no effect.
>> (You can consider Hipe off)
>>
>
> ok, I'd still prefer if you took out that setting.

OK, setting removed. (Test re-done with same result)

>
>
>          {delegate_count, 32},
>>     Why did you set this?
>> I have used this in a try to increase the number of delegate processes
>> used for intra-cluster configuration. Hoping that will increase
>> inter-node communication performance (But this had no effect)
>>
>
> Ditto.
>
>
>  In fact, I am not using any synchronous mechanism. I am publishing
>> messages with no confirm.select.
>> I am using a c client (based on rabbitmq-c) to publish messages with the
>> following config:
>> - Non persistent
>> - No confirm.select
>> - Default Direct exchange
>> - Routing key equal to queue_name
>> - Message size 1KB
>> I am running 10 client processes which are publishing in parallel to 10
>> different queues.
>> Rate is regulated on the client, I can force the publsih rate in the C
>> client.
>>
>> I am running as well 10 other client processes to consume full-speed (no
>> rate regulation) from the 10 queues.
>>
>
> Do the consumers use acknowledgements?

Yes. Consumers are acknowledging the messages as soon as they are received
(No processing in between)
 I have re-done the test with no-ack and I have encountered the same
bottleneck.The rate achieved with no-ack was fluctuating between 18KHZ and
27KHZ (The expected rate is 100KHZ)


>
>  I am using the same setup to publish/consume messages on one node (node
>> 1 where queues are created), with no issue. I can hit up to 100K TPS (5K
>> TPS on each client process).
>> The problem appears only when I try to publish on node1 and consume from
>> node2 (or publish on node2 and consume from node1). The problem appears
>> only when there is an inter-node activity.
>>
>
> So the 100kHz is the aggregate publishing and consuming rate, i.e. 10 x
> 5kHz publishing and 10 x 5kHz consuming?
>
Sorry my mistake, there are 20 processes with 20 queues (Not 10). So 100KHZ
(20 x 5) producing and 100KHZ (20 x 5) consuming

>
> What results do you get in the single-node and multi-node scenarios when
> running the tests with just a single queue and single producer and single
> consumer?
>
> Single producer/consumer/queue on single node max rate achieved: 11KHZ
Single producer/consumer/queue on dual nodes max rate achieved: ~ 11KHZ
The impact of inter-node is not seen here because the traffic is limited by
the single client connection/process. In single consumer/producer
connection, the throughput is limited by the client connection itself not
by the inter-node connection.

Here are below a summary of the tests performed:

Test1: Single node, 20 queues, ACK on
------------------------------------
Nodes: Single node
Producer Processes: 20
Producer process regulate rate: 5KHZ
Acknowledgment: ON
Expected rate: 100 KHZ
Achieved rate: Steady at 100 KHZ
CPU idle: 33%
Note: No bottleneck detected

Test 2: Single node, 20 queues, ACK off
---------------------------------------
Nodes: Single node
Producer Processes: 20
Producer process regulate rate: 5KHZ
Acknowledgment: OFF
Expected rate: 100 KHZ
Achieved rate: Steady at 100 KHZ
CPU idle: 57%
Note: No bottleneck detected

Test 3: Single node, 1 queue, ACK on
------------------------------------
Nodes: Single node
Producer Processes: 1
Producer process regulate rate: 20KHZ
Acknowledgment: ON
Expected rate: 20 KHZ
Achieved rate: Steady 11 KHZ
CPU idle: 94%
Note: Bottleneck on single client connection

Test 4: Single node, 1 queue, ACK off
------------------------------------
Nodes: Single node
Producer Processes: 1
Producer process regulate rate: 20KHZ
Acknowledgment: OFF
Expected rate: 20 KHZ
Achieved rate: Steady 11 KHZ
CPU idle: 96%
Note: Bottleneck on single client connection

Test 5: Dual node, Queues/Producers on same node, 20 queue, ACK on
------------------------------------------------------------------
Nodes: Double node
Queues created on: Node1
Producers on: node 1
Consumers on: node 2
Producer Processes: 20
Producer regulate rate per process: 5KHZ
Acknowledgment: ON
Expected rate: 100KHZ
Achieved rate: Fluctuating between 12KHZ and 21KHZ
CPU idle: 88%
Note: Bottleneck on inter-node connection

Test 6: Dual node, Queues/Producers on same node, 20 queue, ACK off
-------------------------------------------------------------------
Nodes: Double node
Queues created on: Node1
Producers on: node 1
Consumers on: node 2
Producer Processes: 20
Producer regulate rate per process: 5KHZ
Acknowledgment: OFF
Expected rate: 100KHZ
Achieved rate: Fluctuating between 18KHZ and 27KHZ
CPU idle: 91%
Note: Bottleneck on inter-node connection

Test 7: Dual node, Queues/Consumers on same node, 20 queue, ACK on
------------------------------------------------------------------
Nodes: Double node
Queues created on: Node1
Producers on: node 2
Consumers on: node 1
Producer Processes: 20
Producer regulate rate per process: 5KHZ
Acknowledgment: ON
Expected rate: 100KHZ
Achieved rate: Fluctuating between 10KHZ and 20KHZ
CPU idle: 91%
Note: Bottleneck on inter-node connection

Test 8: Dual node, Queues/Consumers on same node, 20 queue, ACK off
-------------------------------------------------------------------
Nodes: Double node
Queues created on: Node1
Producers on: node 2
Consumers on: node 1
Producer Processes: 20
Producer regulate rate per process: 5KHZ
Acknowledgment: OFF
Expected rate: 100KHZ
Achieved rate: Fluctuating between 10KHZ and 20KHZ
CPU idle: 95%
Note: Bottleneck on inter-node connection


Test 9: Dual node, 1 queue, ACK off
------------------------------------
Nodes: Double node
Queues created on: Node1
Producers on: node 2
Consumers on: node 1
Producer Processes: 1
Producer regulate rate per process: 20KHZ
Acknowledgment: OFF
Expected rate: 20KHZ
Achieved rate: Steady 10.5KHZ
CPU idle: 96%
Note: Bottleneck on single client connection

Test 10: Dual node, 1 queue, ACK on
------------------------------------
Nodes: Double node
Queues created on: Node1
Producers on: node 2
Consumers on: node 1
Producer Processes: 1
Producer regulate rate per process: 20KHZ
Acknowledgment: ON
Expected rate: 20KHZ
Achieved rate: Steady 10.5KHZ
CPU idle: 95%
Note: Bottleneck on single client connection


> Regards,
>
> Matthias.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140411/f07f3b89/attachment.html>