Thanks Simon. The 5% figure is useful for me.<br><br>Let me give you a more precise description of what I'm doing to get the 36 message/sec.<br><br><ul><li>RabbitMQ 2.71 on a 3-node cluster with mirrored queues, durable on all nodes.</li>
<li>Client is Python 2.6/Pika 0.9.5.</li><li>Each message publish occurs in a transaction so that we can be sure it's safely in RabbitMQ.</li><li>All nodes are Ubuntu 10.04 VMs with 4GB RAM and 2 or 4 vCPUs.<br></li></ul>
<br>At the heart of things, we're driving a highly complex state machine that manages thousands of VMs and their associated state. Losing track of any state is prohibitively expensive to clean up manually. As such, all state is modeled in clustered databases and/or persistent messages in the message queue. We have to assume that a given client app instance (our management code) may be ungracefully terminated at any moment, so enough state must be modeled to let a new instance pick up and recover. If our database record indicates that a message has been sent, it better darn well be in the hands of the RabbitMQ broker, and not sitting in some Pika client-side queue.<br>
<br>For this reasons, publisher-confirms are not particularly helpful - They assume that the client app will be around to resend the message if the message doesn't get confirmed. Similar story for batching messages. We have to know they've been sent, and we can't stall our state machine waiting for enough message to accumulate to publish multiple messages at once.<br>
<br>My goal in my latest round of experiments is to see what the maximum throughput of a highly available system is in optimal circumstances. We're perfectly willing to spend the money on high end SSDs and networking equipment as necessary.<br>
<br>To prototype what this perf level is, I've configured RabbitMQ with the MNESIA directory pointing to a ramdisk (/tmp). I've configured all the VMs with VMXNET3 networking, am with 16K blocks, are seeing bandwidth of 130MB/sec between VMs in the cluster.<br>
<br>My test app simply writes 6 byte message, one at a time, as quickly as it can. In monitoring the cluster nodes, I'm seeing very low CPU usage, very few writes to the physical disk, and network operation rates of about 700/sec for the master node and 350/sec for the client node.<br>
<br>In short, there's a bottleneck somewhere and it's not obvious where. I'll try your suggestion about replacing tx.commit. Any other insight or guidance would of course be very much appreciated. :-)<br><br>Matt<br>
<br><br><div class="gmail_quote">On Wed, Feb 15, 2012 at 2:08 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com">simon@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 14/02/12 23:07, Matt Pietrek wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm looking to squeeze every last bit of message throughput out of our<br>
mirrored queue setup.<br>
</blockquote>
<br></div>
Hi Matt.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I max out at 36 messages/sec when writing. (Yes,<br>
we use transactions, and we know it's not optimal, but we're valuing<br>
high reliability over speed. Our clients and/or servers could go away at<br>
any moment, so things like Publisher-acknowledge are nice, but take away<br>
from the fundamental disaster hardness.)<br>
<br>
Anyhow, we're running on Ubuntu 10.04 with 2.71 and the default Erlang<br>
install. This gives us R13B03 as the Erlang version.<br>
<br>
Question: Does anybody (and particularly the RabbitMQ folks) have any<br>
input on what sort of perf improvements we might get by switching to a<br>
newer Erlang Version?<br>
</blockquote>
<br></div>
Not much really. We haven't tested extensively but if you got 5% I think you would consider yourself lucky.<br>
<br>
However, 36 msg/s is *really* slow.<br>
<br>
Although mirrored queues are slower than normal ones I would still hope you could stick a couple of zeroes on the end of that figure. So I suspect you are not actually CPU bound at the server.<br>
<br>
I assume when you say "using transactions" you mean publishing a single message per transaction? In that case you are incurring the cost of a network round trip and an fsync() on each publish, and I suspect that is what is killing your performance.<br>
<br>
If I had to guess I would suspect the fsync is the worst of the problem. Assuming the 36 msg/s is from some test app, you might want to remove the tx.select and replace the tx.commit with some meaningless synchronous request (basic.qos is a favourite) and see what happens to your message rate. If it shoots up, the fsync is the problem. If it doesn't, the network round trip is.<br>
<br>
When you have the answer to that, you know whether you're going to want to spend money on fancy networking hardware or SSDs...<br>
<br>
Of course, *really* I would suggest you need to be publishing more messages per round-trip / fsync. So either batch multiple publishes into a transaction or start using confirms instead. I know you say that they "take away from the fundamental disaster hardness" but I'm not sure I understand what you mean by that. Both confirms and tx give you a way to know when a given message is definitely on disc. And in fact tx is implemented in terms of confirms these days.<br>
<br>
Cheers, Simon<span class="HOEnZb"><font color="#888888"><br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
</font></span></blockquote></div><br>