[rabbitmq-discuss] New Persister

[Matthew Sackman]

On Mon, Mar 22, 2010 at 11:26:24AM +1300, Scott Mohekey wrote:
> I'm trying to find out when the new persister will be available.

Me too!

> I've seen
> mention that it was to be ready by July of last year (
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2009-June/003667.html),
> and further mention in November of it not being ready yet (
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2009-November/005386.html
> ).

Yes. Though the "new persister" was in very different states at both of
occasions.

> The reason I ask is that we are attempting to publish ~1000 msgs/sec on a
> single rabbit node, in a persisted queue. The queue is then read by a java
> app on a geographically distant server over the internet, using
> basic.consume. We also require transactional consumption, so currently wrap
> a batch of X (500 for example) messages in a transaction, acking the last
> one (passing true to ack multiple). We have qos set to batch size * 2 (1000
> in the above example).
> 
> We're only managing to get something like 50 or so msgs/sec at the moment.
> The consumption of the messages is slow, but the tx.commit varies in speed
> from a second to many (sometimes even as long as a minute).

Many things can affect this. Are the publishes coming from a single
publisher client or from many clients? Are all the publishes going to
the same queue or to several different queues? How big are the messages
that you're publishing and which client library / language are you
using?

> The version of rabbitmq we currently have installed is 1.7.0. We are
> planning to test with 1.7.2 later today to see if this helps any.
> 
> What are we doing wrong?

The new persister is much better (at least an order of magnitude, and
frequently several, depending on workload) at transactions. I would
suggest you try compiling from source and using that. There are several
people on this list who have been using the new persister with good
success. There are occasionally bugs found, but we tend to fix them
within hours.

There haven't really been any new features added to the new persister
for a few months - it has been solidifying nicely. However, the diff
from the current default branch to the new persister branch (bug21673)
shows the addition of 7000 lines of code. This is the reason why it is
taking a long time to get through QA - not because it's buggy or faults
are being found, but simply because it is a lot of very complex code.

If you do choose to use the new persister, try to ensure the following:
1) Try to make sure nothing else is writing to the disk that Rabbit is
using - Rabbit is pretty good at managing the position of the disk head,
but that tends to go wrong if other applications are writing at the same
time.
2) Ensure plenty of free RAM so the OS can use it as disk cache. This
means a lot of reads can be satisfied without going all the way to disk.
3) You may wish to investigate different file systems - e.g. Matthias's
desktop, Ubuntu, using ext4, gets a pitiful 9 transactions a second,
where as my rather more carefully tuned debian sid system gets 500
txns/sec on an ext3 disk, and 1100 on an ext4 SSD. See the thread at
http://old.nabble.com/limit-number-of-messages-buffered-in-memory-in-new-persister-td27753311.html
4) SSDs can go much faster for txns because their fsync cost is very
very low.
5) Publishing in parallel (several producers, all issuing txns
individually) will get you even better performance.

Matthew