<div dir="ltr">Hi Again,<div><br></div><div>After all the above suggestions, I again try to do my tests with eye on disk usage, RAM usage and network bandwidth utilization. This has now lead to many more questions so could anyone please throw some light on them.</div>
<div><br></div><div>1) I have a quite high bandwidth between my client (producer & receiver) and server. Its around 9.2 Gbps. I have confirmed this by running iperf utility. But the maximum bandwidth I can reach is only 160 mbps. (I have pika client and single rabbitmq DISK type)</div>
<div><br></div><div>2) I then also took network traces using wireshark, and what I observed that the AMQP overhead was quite high then the actual payload. If I send 100 bytes of string then overhead is 300 bytes so for every message of 100 bytes I end up sending 400 bytes on wire. </div>
<div><br></div><div>Since I have publish confirm/ack enabled so I understand part of this but there is also content header and some overhead in content body as well. </div><div>Is this standard for most of the clients or its specific for pika-clients only ? </div>
<div><br></div><div>3) I then also try to observer how my system resources are used. I am new to this but after some googling I learnt a linux command 'dstat -dmnsy'. This provides me disk usage, RAM usage, interrupts and context switches, network send/recev. What I observer that as I increase no of producer/receiver, compare to DISK and RAM usage, its interrupts and context switches, which increases rapidly. </div>
<div><br></div><div>what does this really mean ? </div><div><br></div><div>4) Also, after certain point the value under network send/receive field remains constant. Not sure what does this mean also ?</div><div><br></div>
<div>I am pasting here one of the observed sample from dstat. </div><div><br></div><div><div> mq10@mqserver10:~$ sudo rabbitmqadmin list queues vhost name node messages message_stats.publish_details.rate</div><div>+-------+------+-----------------+----------+------------------------------------+</div>
<div>| vhost | name | node | messages | message_stats.publish_details.rate |</div><div>+-------+------+-----------------+----------+------------------------------------+</div><div>| / | 1 | Moon@mqserver10 | 5 | 3380.4 |</div>
<div>| / | 2 | Moon@mqserver10 | 3 | 3649.0 |</div><div>| / | 3 | Moon@mqserver10 | 6 | 3402.4 |</div><div>| / | 4 | Moon@mqserver10 | 2 | 3450.8 |</div>
<div>| / | 5 | Moon@mqserver10 | 3 | 3444.8 |</div><div>+-------+------+-----------------+----------+------------------------------------+</div><div>mq10@mqserver10:~$ dstat -dmnsy</div>
<div>-dsk/total- ------memory-usage----- -net/total- ----swap--- ---system--</div><div> read writ| used buff cach free| recv send| used free| int csw</div><div> 100k 8974B| 188M 25.7M 115M 9674M| 0 0 | 0 0 | 15k 9759</div>
<div> 0 16k| 189M 25.7M 115M 9673M|6363k 4615k| 0 0 | 22k 6590</div><div> 0 0 | 189M 25.7M 115M 9673M|6666k 4847k| 0 0 | 22k 7846</div><div> 0 0 | 189M 25.7M 115M 9673M|6293k 4500k| 0 0 | 21k 7518</div>
<div> 0 0 | 189M 25.7M 115M 9674M|7049k 5121k| 0 0 | 22k 5936</div><div> 0 0 | 188M 25.7M 115M 9674M|6947k 5081k| 0 0 | 23k 6309</div><div> 0 0 | 188M 25.7M 115M 9674M|6387k 4741k| 0 0 | 22k 7020</div>
<div> 0 0 | 188M 25.7M 115M 9674M|6482k 4739k| 0 0 | 22k 7094</div><div> 0 0 | 189M 25.7M 115M 9673M|5642k 3976k| 0 0 | 18k 7586</div><div> 0 0 | 189M 25.7M 115M 9673M|6246k 4487k| 0 0 | 21k 7497</div>
<div> 0 0 | 189M 25.7M 115M 9673M|6631k 4827k| 0 0 | 21k 5924</div><div> 0 0 | 189M 25.7M 115M 9673M|6615k 4722k| 0 0 | 21k 5448</div><div> 0 0 | 190M 25.7M 115M 9672M|5677k 4051k| 0 0 | 20k 5757</div>
<div> 0 0 | 190M 25.7M 115M 9672M|6007k 4284k| 0 0 | 20k 6581</div><div> 0 0 | 190M 25.7M 115M 9672M|6983k 5052k| 0 0 | 23k 5704</div><div> 0 0 | 190M 25.7M 115M 9672M|6318k 4539k| 0 0 | 22k 6208</div>
<div> 0 0 | 190M 25.7M 115M 9672M|6769k 4862k| 0 0 | 23k 5952</div><div> 0 0 | 189M 25.7M 115M 9673M|6334k 4586k| 0 0 | 22k 6518</div><div> 0 0 | 191M 25.7M 115M 9671M|7019k 5089k| 0 0 | 23k 5937 ^C</div>
<div> </div><div> </div><div> mq10@mqserver10:~$ sudo rabbitmqadmin list queues vhost name node messages message_stats.publish_details.rate</div><div>+-------+------+-----------------+----------+------------------------------------+</div>
<div>| vhost | name | node | messages | message_stats.publish_details.rate |</div><div>+-------+------+-----------------+----------+------------------------------------+</div><div>| / | 1 | Moon@mqserver10 | 3 | 2846.8 |</div>
<div>| / | 2 | Moon@mqserver10 | 6 | 2894.2 |</div><div>| / | 3 | Moon@mqserver10 | 5 | 2950.2 |</div><div>| / | 4 | Moon@mqserver10 | 3 | 2931.8 |</div>
<div>| / | 5 | Moon@mqserver10 | 6 | 2788.2 |</div><div>| / | 6 | Moon@mqserver10 | 7 | 3008.2 |</div><div>+-------+------+-----------------+----------+------------------------------------+</div>
<div>mq10@mqserver10:~$ dstat -dmnsy</div><div>-dsk/total- ------memory-usage----- -net/total- ----swap--- ---system--</div><div> read writ| used buff cach free| recv send| used free| int csw</div><div> 90k 8095B| 189M 25.7M 115M 9673M| 0 0 | 0 0 | 16k 9394</div>
<div> 0 0 | 190M 25.7M 115M 9672M|6893k 4967k| 0 0 | 24k 6852</div><div> 0 0 | 192M 25.7M 115M 9670M|5171k 3601k| 0 0 | 19k 4542</div><div> 0 0 | 191M 25.7M 115M 9671M|7007k 5107k| 0 0 | 24k 6125</div>
<div> 0 0 | 191M 25.7M 115M 9671M|6684k 4890k| 0 0 | 23k 6508</div><div> 0 0 | 191M 25.7M 115M 9671M|6453k 4643k| 0 0 | 23k 6159</div><div> 0 0 | 191M 25.7M 115M 9671M|6342k 4549k| 0 0 | 22k 5418</div>
<div> 0 0 | 191M 25.7M 115M 9671M|6284k 4475k| 0 0 | 22k 5698</div><div> 0 0 | 190M 25.7M 115M 9672M|6783k 4861k| 0 0 | 23k 5781</div><div> 0 0 | 190M 25.7M 115M 9672M|6636k 4707k| 0 0 | 23k 5779</div>
<div> 0 0 | 189M 25.7M 115M 9673M|6807k 4877k| 0 0 | 23k 5590</div><div> 0 0 | 189M 25.7M 115M 9673M|6904k 4949k| 0 0 | 24k 5668</div><div> 0 0 | 189M 25.7M 115M 9673M|7194k 5134k| 0 0 | 24k 5700</div>
<div> 0 0 | 189M 25.7M 115M 9673M|6727k 4779k| 0 0 | 23k 5259</div><div> 0 0 | 189M 25.7M 115M 9673M|7168k 5114k| 0 0 | 25k 5737</div><div> 0 0 | 189M 25.7M 115M 9673M|6943k 5046k| 0 0 | 23k 5646</div>
<div> 0 0 | 189M 25.7M 115M 9674M|6869k 4929k| 0 0 | 23k 5777</div><div> 0 0 | 188M 25.7M 115M 9674M|7037k 5044k| 0 0 | 23k 5586</div><div> 0 0 | 188M 25.7M 115M 9674M|6999k 5031k| 0 0 | 23k 5730</div>
<div> 0 0 | 189M 25.7M 115M 9673M|6599k 4742k| 0 0 | 22k 6504</div><div> 0 0 | 190M 25.7M 115M 9672M|6615k 4834k| 0 0 | 23k 6436</div><div> 0 0 | 190M 25.7M 115M 9672M|6595k 4742k| 0 0 | 23k 5795</div>
<div> </div></div><div><br></div><div>Also, what is the good profiler to run for rabbitmq server ? </div><div><br></div><div>I really want to get hold of how the system resources are used and what is the bottleneck so may be profiler is a good idea. </div>
<div><br></div><div>Please suggest.</div><div><br></div><div>Best Regards,</div><div>Priyanki. </div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Aug 1, 2013 at 9:27 PM, Michael Klishin <span dir="ltr"><<a href="mailto:mklishin@gopivotal.com" target="_blank">mklishin@gopivotal.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Priyanki Vashi:<br>
<div class="im"><br>
> 1) Why there is no linear increase in throughput with DISK type of node as it's seen with RAM type of node ?<br>
<br>
</div>Because disks are much slower than RAM and performing multiple I/O operations in parallel often won't yield<br>
linear increase in I/O throughput (depends on your disks and how they are set up; try with a striping array of SSDs ;))<br>
<br>
With RAM access, it is more likely that you will see nearly linear increase.<br>
<div class="im"><br>
> 3) What can be done to improve throughput in both the Tests ?<br>
<br>
</div>It's hard to give you good advice without seeing the code.<br>
<div class="im"><br>
> 4) Since I have VM with 20 cores dedicated for rabbitMQ execution, how can I load the CPU to it's limit ? with the current tests I can load CPU maximum to 800% with above mentioned throughput. currently the limiting factor seems to be server latency so how to overcome that ?<br>
<br>
</div>I suspect that the limiting factor is disk and network I/O and not CPU.<br>
If Erlang runtime does not need more cores, it will not use more cores.<br>
<br>
Try running iostat or vmstat alongside your benchmarks to see what % of load your disk has.<br>
Monitoring network throughput is another thing to do. Finally, TCP stack configuration matters.<br>
For example, does Pika disable Nagle's algorithm [1] on the sockets it uses? I'm not sure.<br>
<br>
1. <a href="http://boundary.com/blog/2012/05/02/know-a-delay-nagles-algorithm-and-you" target="_blank">http://boundary.com/blog/2012/05/02/know-a-delay-nagles-algorithm-and-you</a><br>
--<br>
MK<br>
<br>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>