<br clear="all">Scott Mohekey<br>Systems/Application Specialist – Fleet – Telogis, Inc.<br><a href="http://www.telogis.com">www.telogis.com</a> <a href="http://www.telogis.co.nz">www.telogis.co.nz</a> <br>+1 949 625-4115 ext. 207 (USA) +64 3339 2825 x207 (NZ)<br>
<br>Leading Global Platform for Location Based Services <br>--<br>This e-mail, and any attachments, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. It is the property of Telogis. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, any attachments thereto, and use of the information contained, is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the original and any copy there of.<br>
<br><br><div class="gmail_quote">On Tue, Mar 23, 2010 at 1:52 AM, Matthew Sackman <span dir="ltr"><<a href="mailto:matthew@lshift.net">matthew@lshift.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On Mon, Mar 22, 2010 at 02:29:52PM +1300, Scott Mohekey wrote:<br>
> On Mon, Mar 22, 2010 at 11:58 AM, Matthew Sackman <<a href="mailto:matthew@lshift.net">matthew@lshift.net</a>>wrote:<br>
</div><div class="im">> We have a single publisher, with 10 production queues which are duplicated<br>
> for testing (so 20 queues all up). All of these queues are persistent.<br>
><br>
> We use a topic exchange, with messages being routed to one or more of the<br>
> queues.<br>
><br>
> The publisher uses the .net client and does no transactions, while the<br>
> consumer uses the java client and wraps batches of messages in transactions.<br>
<br>
</div>Ok, so whilst this is a little different, using the java client, I've<br>
just been testing with:<br>
<br>
rabbitmq-java-client$ sh ./build/dist/runjava.sh com.rabbitmq.examples.MulticastMain -y 0 -s 128 -e e -t fanout -r 1100 -f persistent<br>
<br>
and<br>
<br>
rabbitmq-java-client$ sh build/dist/runjava.sh com/rabbitmq/examples/MulticastMain -x 0 -e e -t fanout -y 10 -n 500 -q 1000<br>
<br>
at the same time. This simulates 10 queues, each with their own<br>
consumer, every consumer sets qos to 1000 and batches up 500 acks in a<br>
transaction before issuing a commit. The publisher publishes 128-byte<br>
messages at a rate of 1100Hz. This means we have 1100 Hz into the<br>
broker, and 11000 Hz out of the server (every publish goes to all 10<br>
queues and hence consumers). My desktop machine does indeed keep up with<br>
this just fine.<br>
<br>
The difference to yous is:<br>
a) You're using topic exchange which will be a little slower.<br>
b) You only have one consumer, which, I assume, is consuming from all<br>
the queues concurrently?<br></blockquote><div><br></div><div>We actually have a consumer per queue, each on a different machine (all of them a considerable distance from the rabbit server).</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
c) My broker and clients are all on the same machine using loopback<br>
interface. This is hardly "geographically distant" :)<br>
<br>
Even if I push the message size up to 4kB, it still copes (though this<br>
is pushing ~100MB/s over loopback. I suspect you probably won't have<br>
this much bandwidth between your consumer and broker).<br>
<div class="im"><br>
> We have our instance of Rabbit running on a XenCenter vm, using shared fibre<br>
> storage. Is this going to be a problem?<br>
<br>
</div>I have no idea, but I'd be surprised if it is. You may wish to use<br>
bonnie++ or similar to see what throughput you can get out of that, but<br>
at least in my tests, it's just bursty up to about 20MB/s (with 4kB<br>
msgs), but mainly under 1MB/s.<br>
<div class="im"><br>
> > 2) Ensure plenty of free RAM so the OS can use it as disk cache. This<br>
> > means a lot of reads can be satisfied without going all the way to disk.<br>
> ><br>
> We have given the vm 8 gigs at the moment.<br>
<br>
</div>That's likely to be enough!<br>
<br>
> We're using ext3.<br>
<br>
Ok, with data=? as the mount option. It may be worth experimenting with<br>
such options just to see whether it makes much difference.<br></blockquote><div><br></div><div>We haven't played with any filesystem tuning yet. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im"><br>
> > 5) Publishing in parallel (several producers, all issuing txns<br>
> > individually) will get you even better performance.<br>
><br>
> Will this make a difference even though we're not using transactions on the<br>
> publishing side?<br>
<br>
</div>No, sorry. I failed to spot in your earlier email that you're only<br>
transacting on consumption, not publish.<br>
<div class="im"><br>
> Also, it seems to me that the slow throughput is on the consumer side.<br>
> Queues have no problem filling up with messages, faster than the consumers<br>
> can keep up with them.<br>
<br>
</div>Indeed. Maybe try to work out what the latency is to your consumer given<br>
the network link you have, what the throughput is, and whether the<br>
messages you're sending are anywhere near that limit? Also try running<br>
the above tests and see what numbers you get out of that. Finally, you<br>
*may* (and I'm totally guessing here) find that you're having issues<br>
with kernel auto-tuning of TCP Buffers. Are all the hosts Linux, or just<br>
the broker?<br>
<font color="#888888"><br>
Matthew<br>
</font></blockquote></div><br><div><br></div><div>One extra thing I should mention, is that while we were seeing this low throughput, half of the queues had an average of 100,000 queued messages. Once we've emptied the queues out, the throughput picks up, but still not much more than around 200 msgs/sec.</div>