[rabbitmq-discuss] Question on RabbitMQ and file I/O characteristics

Tue Aug 6 16:47:11 BST 2013

At this point we probably need to look at what exactly your tests are doing.

You mention that you are publishing in transactions. Does that mean you 
publish each message in a separate transaction? If so then the publishes 
from each channel can't be coalesced together for obvious reasons.

If you are then publishing in multiple channels simultaneously then 
maybe those can coalesce together - although each queue has its own 
journal so if you are publishing to a lot of queues this might not be 
the case. Message contents get written to the (global) message store and 
thus obviously can be coalesced, but the queue index journals are separate.

So:

1) Does that make sense?
2) What does your publishing really look like? And consuming come to that.

Cheers, Simon

PS: for reference, I can publish nearly 4000 persistent confirmed 
messages per second into a single queue at once, with about 90 
writes/sec. So coalescing does happen...

On 05/08/13 18:11, Matt Pietrek wrote:
> Thanks for the background Simon.
>
> One statement caught my attention though:
>
>  > At 40 msg/s you are publishing slowly enough that RabbitMQ probably
> won't start to coalesce these events together.
>
> In my original message I didn't lay out all my results - I was trying to
> keep the question simple. That said, the 4 write/sec pattern held up at
> higher rates. For instance, on a machine with a faster disk, the same
> test could do 400 messages/sec, and per tools like sar, were seeing 1600
> disk writes/sec.
>
> As you can imagine, at those I/O rater we're rapidly coming close to the
> disk's ability to keep up, thus the question about the writes/message rates.
>
> So with this additional context, any additional insight?
>
> Thanks,
>
> Matt
>
>
> On Mon, Aug 5, 2013 at 2:59 AM, Simon MacMullen <simon at rabbitmq.com
> <mailto:simon at rabbitmq.com>> wrote:
>
>     That filename is typical of those of a queue index journal. That's
>     the file that records the state of messages in the queue - when a
>     message has been published, when it's been confirmed (transactional
>     publish is internally implemented in terms of confirms), when it's
>     been delivered, and when it's been acknowledged (I assume you are
>     using acks).
>
>     So it is very possible to get four disk writes to the journal per
>     message for publish / confirm / deliver / ack.
>
>     At 40 msg/s you are publishing slowly enough that RabbitMQ probably
>     won't start to coalesce these events together. So as you start to
>     publish faster you should start to see fewer writes per message -
>     eventually you should see many messages per write.
>
>     Therefore I wouldn't try to scale the numbers you're seeing up to
>     predict anything - you're likely to be much better off running some
>     benchmark with MulticastMain or similar.
>
>     Cheers, Simon
>
>
>     On 03/08/2013 01:08, Matt Pietrek wrote:
>
>         Just want to run this past the RabbitMQ devs and see if this
>         jibes with
>         their expectations.
>
>         Our typical cluster runs RabbitMQ 3.02 on Ubuntu 10.04 on two nodes.
>         All queues are mirrored. All our channels are opened in
>         transactional
>         mode. (We know, this causes things to go slower - That's OK with
>         us, we
>         really want to avoid message loss.)
>
>         What I'm seeing is that for each message written to the broker,
>         we see
>         approximately four disk writes. That is, if the broker is doing 40
>         messages/sec, we see ~160 disk writes. We're getting these
>         number from
>
>         Is this about what should be expected when running this way?
>
>         If it helps, I dug in a bit more to see what the files were. The
>         vast
>         majority of the activity seems to be to files like this:
>
>         msg_store_persistent/743.rdq
>         queues/__E10F54B1OGM7M4LTRX5Z3L4K0/__journal.jif
>
>         Any insight would be very valuable for our planning processes.
>
>         Thanks,
>
>         Matt
>
>
>         _________________________________________________
>         rabbitmq-discuss mailing list
>         rabbitmq-discuss at lists.__rabbitmq.com
>         <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>         https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss
>         <https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>
>
>

-- 
Simon MacMullen
RabbitMQ, Pivotal