<div dir="ltr"><br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">joseph rouphael</b> <span dir="ltr"><<a href="mailto:josephrouphael@gmail.com">josephrouphael@gmail.com</a>></span><br>
Date: Fri, Apr 11, 2014 at 4:11 PM<br>Subject: Re: [rabbitmq-discuss] Performance bottleneck on inter-node connection<br>To: Matthias Radestock <<a href="mailto:matthias@rabbitmq.com">matthias@rabbitmq.com</a>><br><br>
<br><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote"><div class="">On Thu, Apr 10, 2014 at 7:01 PM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">(re-adding the list)<div><br>
<br>
On 10/04/14 16:42, joseph rouphael wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
On Thu, Apr 10, 2014 at 11:57 AM, Matthias Radestock<br></div><div>
<<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a> <mailto:<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>><u></u>> wrote:<br>
Have you tried this w/o hipe?<br>
In fact, HIPE is not enabled in Erlang, so this config has no effect.<br>
(You can consider Hipe off)<br>
</div></blockquote>
<br>
ok, I'd still prefer if you took out that setting.</blockquote></div><div>OK, setting removed. (Test re-done with same result)</div><div class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
{delegate_count, 32},<br>
Why did you set this?<br>
I have used this in a try to increase the number of delegate processes<br>
used for intra-cluster configuration. Hoping that will increase<br>
inter-node communication performance (But this had no effect)<br>
</blockquote>
<br></div>
Ditto.<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In fact, I am not using any synchronous mechanism. I am publishing<br>
messages with no confirm.select.<br>
I am using a c client (based on rabbitmq-c) to publish messages with the<br>
following config:<br>
- Non persistent<br>
- No confirm.select<br>
- Default Direct exchange<br>
- Routing key equal to queue_name<br>
- Message size 1KB<br>
I am running 10 client processes which are publishing in parallel to 10<br>
different queues.<br>
Rate is regulated on the client, I can force the publsih rate in the C<br>
client.<br>
<br>
I am running as well 10 other client processes to consume full-speed (no<br>
rate regulation) from the 10 queues.<br>
</blockquote>
<br></div>
Do the consumers use acknowledgements?</blockquote></div><div>Yes. Consumers are acknowledging the messages as soon as they are received (No processing in between)</div><div> I have re-done the test with no-ack and I have encountered the same bottleneck.The rate achieved with no-ack was fluctuating between 18KHZ and 27KHZ (The expected rate is 100KHZ)</div>
<div class="">
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I am using the same setup to publish/consume messages on one node (node<br>
1 where queues are created), with no issue. I can hit up to 100K TPS (5K<br>
TPS on each client process).<br>
The problem appears only when I try to publish on node1 and consume from<br>
node2 (or publish on node2 and consume from node1). The problem appears<br>
only when there is an inter-node activity.<br>
</blockquote>
<br></div>
So the 100kHz is the aggregate publishing and consuming rate, i.e. 10 x 5kHz publishing and 10 x 5kHz consuming?<br></blockquote></div><div>Sorry my mistake, there are 20 processes with 20 queues (Not 10). So 100KHZ (20 x 5) producing and 100KHZ (20 x 5) consuming </div>
<div class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
What results do you get in the single-node and multi-node scenarios when running the tests with just a single queue and single producer and single consumer?<br>
<br></blockquote></div><div>Single producer/consumer/queue on single node max rate achieved: 11KHZ </div><div>Single producer/consumer/queue on dual nodes max rate achieved: ~ 11KHZ </div>
<div>The impact of inter-node is not seen here because the traffic is limited by the single client connection/process. In single consumer/producer connection, the throughput is limited by the client connection itself not by the inter-node connection. </div>
<div><br></div><div>Here are below a summary of the tests performed:</div><div><br></div><div><div>Test1: Single node, 20 queues, ACK on</div><div>------------------------------------</div><div>Nodes: Single node</div><div>
Producer Processes: 20</div><div>Producer process regulate rate: 5KHZ</div><div>Acknowledgment: ON</div><div>Expected rate: 100 KHZ</div><div>Achieved rate: Steady at 100 KHZ</div><div>CPU idle: 33%</div><div>Note: No bottleneck detected</div>
</div><div><br></div><div><div>Test 2: Single node, 20 queues, ACK off</div><div>---------------------------------------</div><div>Nodes: Single node</div><div>Producer Processes: 20</div><div>Producer process regulate rate: 5KHZ</div>
<div>Acknowledgment: OFF</div><div>Expected rate: 100 KHZ</div><div>Achieved rate: Steady at 100 KHZ</div><div>CPU idle: 57%</div><div>Note: No bottleneck detected</div><div><br></div><div>Test 3: Single node, 1 queue, ACK on</div>
<div>------------------------------------</div><div>Nodes: Single node</div><div>Producer Processes: 1</div><div>Producer process regulate rate: 20KHZ</div><div>Acknowledgment: ON</div><div>Expected rate: 20 KHZ</div><div>
Achieved rate: Steady 11 KHZ</div><div>CPU idle: 94%</div><div>Note: Bottleneck on single client connection</div></div><div><br></div><div><div>Test 4: Single node, 1 queue, ACK off</div><div>------------------------------------</div>
<div>Nodes: Single node</div><div>Producer Processes: 1</div><div>Producer process regulate rate: 20KHZ</div><div>Acknowledgment: OFF</div><div>Expected rate: 20 KHZ</div><div>Achieved rate: Steady 11 KHZ</div><div>CPU idle: 96%</div>
<div>Note: Bottleneck on single client connection</div><div><br></div><div>Test 5: Dual node, Queues/Producers on same node, 20 queue, ACK on</div><div>------------------------------------------------------------------</div>
<div>Nodes: Double node</div><div>Queues created on: Node1</div><div>Producers on: node 1</div><div>Consumers on: node 2</div><div>Producer Processes: 20</div><div>Producer regulate rate per process: 5KHZ</div><div>Acknowledgment: ON</div>
<div>Expected rate: 100KHZ</div><div>Achieved rate: Fluctuating between 12KHZ and 21KHZ</div><div>CPU idle: 88%</div><div>Note: Bottleneck on inter-node connection</div></div><div><br></div><div><div>Test 6: Dual node, Queues/Producers on same node, 20 queue, ACK off</div>
<div>-------------------------------------------------------------------</div><div>Nodes: Double node</div><div>Queues created on: Node1</div><div>Producers on: node 1</div><div>Consumers on: node 2</div><div>Producer Processes: 20</div>
<div>Producer regulate rate per process: 5KHZ</div><div>Acknowledgment: OFF</div><div>Expected rate: 100KHZ</div><div>Achieved rate: Fluctuating between 18KHZ and 27KHZ</div><div>CPU idle: 91%</div><div>Note: Bottleneck on inter-node connection</div>
<div><br></div><div>Test 7: Dual node, Queues/Consumers on same node, 20 queue, ACK on</div><div>------------------------------------------------------------------</div><div>Nodes: Double node</div><div>Queues created on: Node1</div>
<div>Producers on: node 2</div><div>Consumers on: node 1</div><div>Producer Processes: 20</div><div>Producer regulate rate per process: 5KHZ</div><div>Acknowledgment: ON</div><div>Expected rate: 100KHZ</div><div>Achieved rate: Fluctuating between 10KHZ and 20KHZ</div>
<div>CPU idle: 91%</div><div>Note: Bottleneck on inter-node connection</div></div><div><br></div><div><div>Test 8: Dual node, Queues/Consumers on same node, 20 queue, ACK off</div><div>-------------------------------------------------------------------</div>
<div>Nodes: Double node</div><div>Queues created on: Node1</div><div>Producers on: node 2</div><div>Consumers on: node 1</div><div>Producer Processes: 20</div><div>Producer regulate rate per process: 5KHZ</div><div>Acknowledgment: OFF</div>
<div>Expected rate: 100KHZ</div><div>Achieved rate: Fluctuating between 10KHZ and 20KHZ</div><div>CPU idle: 95%</div><div>Note: Bottleneck on inter-node connection</div><div><br></div><div><br></div><div>Test 9: Dual node, 1 queue, ACK off</div>
<div>------------------------------------</div><div>Nodes: Double node</div><div>Queues created on: Node1</div><div>Producers on: node 2</div><div>Consumers on: node 1</div><div>Producer Processes: 1</div><div>Producer regulate rate per process: 20KHZ</div>
<div>Acknowledgment: OFF</div><div>Expected rate: 20KHZ</div><div>Achieved rate: Steady 10.5KHZ</div><div>CPU idle: 96%</div><div>Note: Bottleneck on single client connection</div></div><div><br></div><div><div>Test 10: Dual node, 1 queue, ACK on</div>
<div>------------------------------------</div><div>Nodes: Double node</div><div>Queues created on: Node1</div><div>Producers on: node 2</div><div>Consumers on: node 1</div><div>Producer Processes: 1</div><div>Producer regulate rate per process: 20KHZ</div>
<div>Acknowledgment: ON</div><div>Expected rate: 20KHZ</div><div>Achieved rate: Steady 10.5KHZ</div><div>CPU idle: 95%</div><div>Note: Bottleneck on single client connection</div></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Regards,<br>
<br>
Matthias.<br>
</blockquote></div><br></div></div>
</div><br></div>