<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hello, Alvaro.<div><br></div><div>Thank you for the advise. I had already checked that. Disk IO seems to be fine and VM is slightly loaded. Looks like there is a bottleneck and it's visible surprisingly only when Topic Exchange is used. I will continue digging. If I find an explanation I will let the community know. Initially this strange behavior was totally mystery since I didn't think I could have it because of Virtualization :)</div><div><br></div><div>Thank you,</div><div>Konstantin.</div><div><br></div><div><div><div>On May 7, 2012, at 3:29 PM, Alvaro Videla wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<div>
Have you checked Disk I/O performance? </div><div><br></div><div>A binding on a topic exchange has a bit more entries in Mnesia than the other exchanges.
</div><div><br></div><div>Regards,</div><div><br></div><div>Alvaro</div>
<div><div><br></div><div>-- </div><div>Sent with <a href="http://www.sparrowmailapp.com/?sig">Sparrow</a></div><div><br></div></div><p style="color: #A0A0A8;">On Tuesday, May 8, 2012 at 12:24 AM, Konstantin Kalin wrote:</p>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div><div>Hello, Simon.</div><div><br></div><div>Finally I found an answer why we had different results with same test code. Something is wrong between KVM virtualization and RabbitMQ when Topic Exchange is used :) At least once I started RabbitMQ on physical server I get understandable results: 90k on Direct Exchange and 75k on Topic exchange. I have checked Network link for RabbitMQ VM and it doesn't look bad. I will continue digging around to find an explanation why it happens. If you can provide any additional advice I will really appreciate it. </div><div><br></div><div>Thank you,</div><div>Konstantin. </div><div><br></div><div>On May 4, 2012, at 1:51 PM, Konstantin Kalin wrote:</div><div><br></div><blockquote type="cite"><div><div>Hello, Simon.</div><div><br></div><div>Thank you for your help here. See inline. </div><div><br></div><div>On May 4, 2012, at 3:11 AM, Simon MacMullen wrote:</div><div><br></div><blockquote type="cite"><div><div> amqp_channel:cast(Channel, Publish, Msg).</div><div><br></div><div>Since this is pushing messages as fast as possible into the client channel process, you end up filling its mailbox as you can give it messages faster than it can send them out on the wire.</div></div></blockquote><div><br></div><div>Yes. I know about this and this is why I created rate control on Client side. I thought that amqp_channel:cast(Channel, Publish, Msg) is better way to create a load :) Well, I have changed it to amqp_channel:call. Thank you for the advice. </div><div><br></div><div><br></div><blockquote type="cite"><div><div>direct, 1 producer, 1 consumer: ~25kHz</div><div>topic, 1 producer, 1 consumer: ~14kHz</div><div>direct, 10 producers, 10 consumers: ~25kHz</div><div> (but queues grew since consumers could not keep up)</div><div>topic, 10 producers, 10 consumers: ~16kHz</div><div><br></div><div>on my workstation. This is very roughly what I would expect - with tiny messages quite a lot of the cost of each message is routing, and topic routing *is* more complex.</div></div></blockquote><div><br></div><div>Well. It looks much better than I have :) And this is why I have asked to guide me what I'm doing wrong. I believe that RabbitMQ is great product and this is why I spend time on different tests to gain better understanding how it works. Thus I will use RabbitMQ more effectively. </div><div> </div><blockquote type="cite"><div>So I'm afraid I don't really trust the results you got - how were you choosing a rate limit for the client? Too low and you're not driving the broker as hard as you can, and with that cast in place, too high and you'll start to eat all memory and eventually cause RabbitMQ to get swapped out.</div></blockquote><div>Understood. Usually the load test results causes a lot of confusions and questions since people makes load-tests for particular cases. </div><div>Please don't treat me wrong. I don't blame RabbitMQ. I rather will blame me for these strange results. </div><div><br></div><div>Let me explain my test setup:</div><div>I have two physical servers: 16 AuthenticAMD CPU, 16 Gb memory each. I use one physical server to virtualize RabbitMQ using KVM. RabbitMQ has 8 CPU and 4Gb. And there are no other VMs running on this physical server. And I use second physical server to run Consumer/Publisher code that you tried. </div><div><br></div><div>I repeated the test with direct and topic exchanges after I modified the code as mentioted above. </div><div>10 consumers and publishers for each type of exchange. I started publishing using following commands:</div><div>Test 1 - publisher:start_publish(<<"kkalin_test1">>, <<"topic">>, 20000, 20000, 10, 0).</div><div>Test 2 - publisher:start_publish(<<"kkalin_test">>, <<"direct">>, 20000, 20000, 10, 0). </div><div><br></div><div>I got: ~4k for Topic and ~60k for Direct :) See attached screenshots. I don't understand why. And I think it's hard to blame that publisher_wrk works slow for Topic exchange. There is no dedicated logic in software code that can affect messages rate depending on Exchange type. Same code, same machines and results with big differences. :(</div><div><br></div><div>May be you can provide a few advices what I can check to understand where is a bottle neck? </div><div>According to vmstat and htop, RabbitMQ and Client servers were not loaded in case of Topic Exchange. Looks like that RabbitMQ and Publisher are waiting each other. </div><div><br></div><div>Thank you for patience,</div><div>Kostya. </div><div><br></div><div><Screen Shot 2012-05-04 at 12.57.24 PM.png></div><div><Screen Shot 2012-05-04 at 1.09.10 PM.png></div></div></blockquote><div><br></div><div>_______________________________________________</div><div>rabbitmq-discuss mailing list</div><div><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a></div><div><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>
</blockquote></div><br></div></body></html>