[rabbitmq-discuss] New Persister

[Matthew Sackman]

On Mon, Mar 22, 2010 at 02:29:52PM +1300, Scott Mohekey wrote:
> On Mon, Mar 22, 2010 at 11:58 AM, Matthew Sackman <matthew at lshift.net>wrote:
> We have a single publisher, with 10 production queues which are duplicated
> for testing (so 20 queues all up). All of these queues are persistent.
> 
> We use a topic exchange, with messages being routed to one or more of the
> queues.
> 
> The publisher uses the .net client and does no transactions, while the
> consumer uses the java client and wraps batches of messages in transactions.

Ok, so whilst this is a little different, using the java client, I've
just been testing with:

rabbitmq-java-client$ sh ./build/dist/runjava.sh com.rabbitmq.examples.MulticastMain -y 0 -s 128 -e e -t fanout -r 1100 -f persistent

and

rabbitmq-java-client$ sh build/dist/runjava.sh com/rabbitmq/examples/MulticastMain -x 0 -e e -t fanout -y 10 -n 500 -q 1000

at the same time. This simulates 10 queues, each with their own
consumer, every consumer sets qos to 1000 and batches up 500 acks in a
transaction before issuing a commit. The publisher publishes 128-byte
messages at a rate of 1100Hz. This means we have 1100 Hz into the
broker, and 11000 Hz out of the server (every publish goes to all 10
queues and hence consumers). My desktop machine does indeed keep up with
this just fine.

The difference to yous is:
a) You're using topic exchange which will be a little slower.
b) You only have one consumer, which, I assume, is consuming from all
the queues concurrently?
c) My broker and clients are all on the same machine using loopback
interface. This is hardly "geographically distant" :)

Even if I push the message size up to 4kB, it still copes (though this
is pushing ~100MB/s over loopback. I suspect you probably won't have
this much bandwidth between your consumer and broker).

> We have our instance of Rabbit running on a XenCenter vm, using shared fibre
> storage. Is this going to be a problem?

I have no idea, but I'd be surprised if it is. You may wish to use
bonnie++ or similar to see what throughput you can get out of that, but
at least in my tests, it's just bursty up to about 20MB/s (with 4kB
msgs), but mainly under 1MB/s.

> > 2) Ensure plenty of free RAM so the OS can use it as disk cache. This
> > means a lot of reads can be satisfied without going all the way to disk.
> >
> We have given the vm 8 gigs at the moment.

That's likely to be enough!

> We're using ext3.

Ok, with data=? as the mount option. It may be worth experimenting with
such options just to see whether it makes much difference.

> > 5) Publishing in parallel (several producers, all issuing txns
> > individually) will get you even better performance.
> 
> Will this make a difference even though we're not using transactions on the
> publishing side?

No, sorry. I failed to spot in your earlier email that you're only
transacting on consumption, not publish.

> Also, it seems to me that the slow throughput is on the consumer side.
> Queues have no problem filling up with messages, faster than the consumers
> can keep up with them.

Indeed. Maybe try to work out what the latency is to your consumer given
the network link you have, what the throughput is, and whether the
messages you're sending are anywhere near that limit? Also try running
the above tests and see what numbers you get out of that. Finally, you
*may* (and I'm totally guessing here) find that you're having issues
with kernel auto-tuning of TCP Buffers. Are all the hosts Linux, or just
the broker?

Matthew