[rabbitmq-discuss] New Persister

[Scott Mohekey]

Reply inline below:

Scott Mohekey
Systems/Application Specialist – Fleet – Telogis, Inc.
www.telogis.com  www.telogis.co.nz
+1 949 625-4115 ext. 207 (USA)  +64 3339 2825 x207 (NZ)

Leading Global Platform for Location Based Services
--
This e-mail, and any attachments, is intended only for use by the
addressee(s) named herein and may contain legally privileged and/or
confidential information.  It is the property of Telogis.  If you are not
the intended recipient of this e-mail, you are hereby notified that any
dissemination, distribution or copying of this e-mail, any attachments
thereto, and use of the information contained, is strictly prohibited.  If
you have received this e-mail in error, please notify the sender and
permanently delete the original and any copy there of.


On Mon, Mar 22, 2010 at 11:58 AM, Matthew Sackman <matthew at lshift.net>wrote:

> On Mon, Mar 22, 2010 at 11:26:24AM +1300, Scott Mohekey wrote:
> > I'm trying to find out when the new persister will be available.
>
> Me too!
>
> > I've seen
> > mention that it was to be ready by July of last year (
> >
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2009-June/003667.html
> ),
> > and further mention in November of it not being ready yet (
> >
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2009-November/005386.html
> > ).
>
> Yes. Though the "new persister" was in very different states at both of
> occasions.
>
> > The reason I ask is that we are attempting to publish ~1000 msgs/sec on a
> > single rabbit node, in a persisted queue. The queue is then read by a
> java
> > app on a geographically distant server over the internet, using
> > basic.consume. We also require transactional consumption, so currently
> wrap
> > a batch of X (500 for example) messages in a transaction, acking the last
> > one (passing true to ack multiple). We have qos set to batch size * 2
> (1000
> > in the above example).
> >
> > We're only managing to get something like 50 or so msgs/sec at the
> moment.
> > The consumption of the messages is slow, but the tx.commit varies in
> speed
> > from a second to many (sometimes even as long as a minute).
>
> Many things can affect this. Are the publishes coming from a single
> publisher client or from many clients? Are all the publishes going to
> the same queue or to several different queues? How big are the messages
> that you're publishing and which client library / language are you
> using?
>
>
We have a single publisher, with 10 production queues which are duplicated
for testing (so 20 queues all up). All of these queues are persistent.

We use a topic exchange, with messages being routed to one or more of the
queues.

The publisher uses the .net client and does no transactions, while the
consumer uses the java client and wraps batches of messages in transactions.


> > The version of rabbitmq we currently have installed is 1.7.0. We are
> > planning to test with 1.7.2 later today to see if this helps any.
> >
> > What are we doing wrong?
>
> The new persister is much better (at least an order of magnitude, and
> frequently several, depending on workload) at transactions. I would
> suggest you try compiling from source and using that. There are several
> people on this list who have been using the new persister with good
> success. There are occasionally bugs found, but we tend to fix them
> within hours.
>
> There haven't really been any new features added to the new persister
> for a few months - it has been solidifying nicely. However, the diff
> from the current default branch to the new persister branch (bug21673)
> shows the addition of 7000 lines of code. This is the reason why it is
> taking a long time to get through QA - not because it's buggy or faults
> are being found, but simply because it is a lot of very complex code.
>
> If you do choose to use the new persister, try to ensure the following:
> 1) Try to make sure nothing else is writing to the disk that Rabbit is
> using - Rabbit is pretty good at managing the position of the disk head,
> but that tends to go wrong if other applications are writing at the same
> time.
>

We have our instance of Rabbit running on a XenCenter vm, using shared fibre
storage. Is this going to be a problem?


> 2) Ensure plenty of free RAM so the OS can use it as disk cache. This
> means a lot of reads can be satisfied without going all the way to disk.
>
We have given the vm 8 gigs at the moment.


> 3) You may wish to investigate different file systems - e.g. Matthias's
> desktop, Ubuntu, using ext4, gets a pitiful 9 transactions a second,
> where as my rather more carefully tuned debian sid system gets 500
> txns/sec on an ext3 disk, and 1100 on an ext4 SSD. See the thread at
>
> http://old.nabble.com/limit-number-of-messages-buffered-in-memory-in-new-persister-td27753311.html


We're using ext3.


>
> 4) SSDs can go much faster for txns because their fsync cost is very
> very low.
> 5) Publishing in parallel (several producers, all issuing txns
> individually) will get you even better performance.
>

Will this make a difference even though we're not using transactions on the
publishing side?

Also, it seems to me that the slow throughput is on the consumer side.
Queues have no problem filling up with messages, faster than the consumers
can keep up with them.

Matthew
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100322/3c9bfea6/attachment.htm