[rabbitmq-discuss] Question on RabbitMQ and file I/O characteristics

Tue Aug 6 18:42:54 BST 2013

1) Does that make sense?

For the most part, yes. Thank you.

2) What does your publishing really look like? And consuming come to that.

So the app is something I call "echotest". It spins up two ioloops in
separate threads ('main', 'echo'). Each ioloop thread listens on its own
queue.

The "on_message" methods in both 'main' and 'echo' do effectively the
following:

- Write a new message to the other queue
- Acknowledge the incoming message

The net effect is that messages 'ping-pong' between the two ioloop threads
as fast as possible. In the 'main' ioloop init, it sends an initial message
to kick things off.

I've added counters and timing logic to calculate the current message rate.

In terms of the queue config, the queues are declared like this:

    channel.queue_declare(queue=echo_queue_name, durable=True,
arguments={"x-ha-policy": "all"})

Broker config is two clustered 3.0.2 nodes.

For writing messages, it looks like this:

    channel.basic_publish(exchange='',
        routing_key=echo_queue_name,
        body=json.dumps(payload_dict),
        mandatory=True,
        properties=BasicProperties(delivery_mode=2))

I also have a global switch (USE_TRANSACTIONS) that lets me switch between
using transactions and not. When set, it adds tx_select() when opening the
channels and tx_commit after publishing messages. For the tests I was
running, I wasn't using transactions. If I do run with transactions the
numbers are very similar (surprising I know, but I stepped through the code
to verify that the tx_xxx calls are only made when I intend them to be.)

I absolutely realize this particular setup isn't optimal for driving high
broker load. For the moment, I'm more interested in how many "RPC"-style
calls/second we could expect between two given endpoints.

Thanks for your help in digging into this!

Matt

On Tue, Aug 6, 2013 at 8:47 AM, Simon MacMullen <simon at rabbitmq.com> wrote:

> At this point we probably need to look at what exactly your tests are
> doing.
>
> You mention that you are publishing in transactions. Does that mean you
> publish each message in a separate transaction? If so then the publishes
> from each channel can't be coalesced together for obvious reasons.
>
> If you are then publishing in multiple channels simultaneously then maybe
> those can coalesce together - although each queue has its own journal so if
> you are publishing to a lot of queues this might not be the case. Message
> contents get written to the (global) message store and thus obviously can
> be coalesced, but the queue index journals are separate.
>
> So:
>
> 1) Does that make sense?
> 2) What does your publishing really look like? And consuming come to that.
>
> Cheers, Simon
>
> PS: for reference, I can publish nearly 4000 persistent confirmed messages
> per second into a single queue at once, with about 90 writes/sec. So
> coalescing does happen...
>
>
> On 05/08/13 18:11, Matt Pietrek wrote:
>
>> Thanks for the background Simon.
>>
>> One statement caught my attention though:
>>
>>  > At 40 msg/s you are publishing slowly enough that RabbitMQ probably
>> won't start to coalesce these events together.
>>
>> In my original message I didn't lay out all my results - I was trying to
>> keep the question simple. That said, the 4 write/sec pattern held up at
>> higher rates. For instance, on a machine with a faster disk, the same
>> test could do 400 messages/sec, and per tools like sar, were seeing 1600
>> disk writes/sec.
>>
>> As you can imagine, at those I/O rater we're rapidly coming close to the
>> disk's ability to keep up, thus the question about the writes/message
>> rates.
>>
>> So with this additional context, any additional insight?
>>
>> Thanks,
>>
>> Matt
>>
>>
>> On Mon, Aug 5, 2013 at 2:59 AM, Simon MacMullen <simon at rabbitmq.com
>> <mailto:simon at rabbitmq.com>> wrote:
>>
>>     That filename is typical of those of a queue index journal. That's
>>     the file that records the state of messages in the queue - when a
>>     message has been published, when it's been confirmed (transactional
>>     publish is internally implemented in terms of confirms), when it's
>>     been delivered, and when it's been acknowledged (I assume you are
>>     using acks).
>>
>>     So it is very possible to get four disk writes to the journal per
>>     message for publish / confirm / deliver / ack.
>>
>>     At 40 msg/s you are publishing slowly enough that RabbitMQ probably
>>     won't start to coalesce these events together. So as you start to
>>     publish faster you should start to see fewer writes per message -
>>     eventually you should see many messages per write.
>>
>>     Therefore I wouldn't try to scale the numbers you're seeing up to
>>     predict anything - you're likely to be much better off running some
>>     benchmark with MulticastMain or similar.
>>
>>     Cheers, Simon
>>
>>
>>     On 03/08/2013 01:08, Matt Pietrek wrote:
>>
>>         Just want to run this past the RabbitMQ devs and see if this
>>         jibes with
>>         their expectations.
>>
>>         Our typical cluster runs RabbitMQ 3.02 on Ubuntu 10.04 on two
>> nodes.
>>         All queues are mirrored. All our channels are opened in
>>         transactional
>>         mode. (We know, this causes things to go slower - That's OK with
>>         us, we
>>         really want to avoid message loss.)
>>
>>         What I'm seeing is that for each message written to the broker,
>>         we see
>>         approximately four disk writes. That is, if the broker is doing 40
>>         messages/sec, we see ~160 disk writes. We're getting these
>>         number from
>>
>>         Is this about what should be expected when running this way?
>>
>>         If it helps, I dug in a bit more to see what the files were. The
>>         vast
>>         majority of the activity seems to be to files like this:
>>
>>         msg_store_persistent/743.rdq
>>         queues/__**E10F54B1OGM7M4LTRX5Z3L4K0/__**journal.jif
>>
>>
>>         Any insight would be very valuable for our planning processes.
>>
>>         Thanks,
>>
>>         Matt
>>
>>
>>         ______________________________**___________________
>>         rabbitmq-discuss mailing list
>>         rabbitmq-discuss at lists.__rabbi**tmq.com <http://rabbitmq.com>
>>         <mailto:rabbitmq-discuss@**lists.rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
>> >
>>         https://lists.rabbitmq.com/__**cgi-bin/mailman/listinfo/__**
>> rabbitmq-discuss<https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss>
>>         <https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**
>> rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>> >
>>
>>
>>
>>
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130806/f69f68f1/attachment.htm>