<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">1) Does that make sense?</span><br><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">For the most part, yes. Thank you.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">2) What does your publishing really look like? And consuming come to that.</span><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">So the app is something I call "echotest". It spins up two ioloops in separate threads ('main', 'echo'). Each ioloop thread listens on its own queue.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">The "on_message" methods in both 'main' and 'echo' do effectively the following:</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">- Write a new message to the other queue</span></div><div><span style="font-family:arial,sans-serif;font-size:13px">- Acknowledge the incoming message</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><font face="arial, sans-serif">The net effect is that messages 'ping-pong' between the two ioloop threads as fast as possible. In the 'main' ioloop init, it sends an initial message to kick things off.</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">I've added counters and timing logic to calculate the current message rate.</font></div><div><font face="arial, sans-serif"><br>
</font></div><div><font face="arial, sans-serif">In terms of the queue config, the queues are declared like this:</font></div><div><font face="arial, sans-serif"><br></font></div><div><div>� � channel.queue_declare(queue=echo_queue_name, durable=True, arguments={"x-ha-policy": "all"})</div>
</div><div><br></div><div>Broker config is two clustered 3.0.2 nodes.</div><div><br></div><div>For writing messages, it looks like this:</div><div><br></div><div><div>� � channel.basic_publish(exchange='',</div><div>
� � � � routing_key=echo_queue_name,</div><div>� � � � body=json.dumps(payload_dict),</div><div>� � � � mandatory=True,</div><div>� � � � properties=BasicProperties(delivery_mode=2))</div></div><div><br></div><div>I also have a global switch (USE_TRANSACTIONS) that lets me switch between using transactions and not. When set, it adds tx_select() when opening the channels and tx_commit after publishing messages. For the tests I was running, I wasn't using transactions. If I do run with transactions the numbers are very similar (surprising I know, but I stepped through the code to verify that the tx_xxx calls are only made when I intend them to be.)</div>
<div><br></div><div>I absolutely realize this particular setup isn't optimal for driving high broker load. For the moment, I'm more interested in how many "RPC"-style calls/second we could expect between two given endpoints.</div>
<div><br></div><div>Thanks for your help in digging into this!</div><div><br></div><div>Matt</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 6, 2013 at 8:47 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">At this point we probably need to look at what exactly your tests are doing.<br>
<br>
You mention that you are publishing in transactions. Does that mean you publish each message in a separate transaction? If so then the publishes from each channel can't be coalesced together for obvious reasons.<br>
<br>
If you are then publishing in multiple channels simultaneously then maybe those can coalesce together - although each queue has its own journal so if you are publishing to a lot of queues this might not be the case. Message contents get written to the (global) message store and thus obviously can be coalesced, but the queue index journals are separate.<br>
<br>
So:<br>
<br>
1) Does that make sense?<br>
2) What does your publishing really look like? And consuming come to that.<br>
<br>
Cheers, Simon<br>
<br>
PS: for reference, I can publish nearly 4000 persistent confirmed messages per second into a single queue at once, with about 90 writes/sec. So coalescing does happen...<div class="im"><br>
<br>
On 05/08/13 18:11, Matt Pietrek wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
Thanks for the background Simon.<br>
<br>
One statement caught my attention though:<br>
<br>
�> At 40 msg/s you are publishing slowly enough that RabbitMQ probably<br>
won't start to coalesce these events together.<br>
<br>
In my original message I didn't lay out all my results - I was trying to<br>
keep the question simple. That said, the 4 write/sec pattern held up at<br>
higher rates. For instance, on a machine with a faster disk, the same<br>
test could do 400 messages/sec, and per tools like sar, were seeing 1600<br>
disk writes/sec.<br>
<br>
As you can imagine, at those I/O rater we're rapidly coming close to the<br>
disk's ability to keep up, thus the question about the writes/message rates.<br>
<br>
So with this additional context, any additional insight?<br>
<br>
Thanks,<br>
<br>
Matt<br>
<br>
<br>
On Mon, Aug 5, 2013 at 2:59 AM, Simon MacMullen <<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a><br></div><div><div class="h5">
<mailto:<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>>> wrote:<br>
<br>
� � That filename is typical of those of a queue index journal. That's<br>
� � the file that records the state of messages in the queue - when a<br>
� � message has been published, when it's been confirmed (transactional<br>
� � publish is internally implemented in terms of confirms), when it's<br>
� � been delivered, and when it's been acknowledged (I assume you are<br>
� � using acks).<br>
<br>
� � So it is very possible to get four disk writes to the journal per<br>
� � message for publish / confirm / deliver / ack.<br>
<br>
� � At 40 msg/s you are publishing slowly enough that RabbitMQ probably<br>
� � won't start to coalesce these events together. So as you start to<br>
� � publish faster you should start to see fewer writes per message -<br>
� � eventually you should see many messages per write.<br>
<br>
� � Therefore I wouldn't try to scale the numbers you're seeing up to<br>
� � predict anything - you're likely to be much better off running some<br>
� � benchmark with MulticastMain or similar.<br>
<br>
� � Cheers, Simon<br>
<br>
<br>
� � On 03/08/2013 01:08, Matt Pietrek wrote:<br>
<br>
� � � � Just want to run this past the RabbitMQ devs and see if this<br>
� � � � jibes with<br>
� � � � their expectations.<br>
<br>
� � � � Our typical cluster runs RabbitMQ 3.02 on Ubuntu 10.04 on two nodes.<br>
� � � � All queues are mirrored. All our channels are opened in<br>
� � � � transactional<br>
� � � � mode. (We know, this causes things to go slower - That's OK with<br>
� � � � us, we<br>
� � � � really want to avoid message loss.)<br>
<br>
� � � � What I'm seeing is that for each message written to the broker,<br>
� � � � we see<br>
� � � � approximately four disk writes. That is, if the broker is doing 40<br>
� � � � messages/sec, we see ~160 disk writes. We're getting these<br>
� � � � number from<br>
<br>
� � � � Is this about what should be expected when running this way?<br>
<br>
� � � � If it helps, I dug in a bit more to see what the files were. The<br>
� � � � vast<br>
� � � � majority of the activity seems to be to files like this:<br>
<br>
� � � � msg_store_persistent/743.rdq<br></div></div>
� � � � queues/__<u></u>E10F54B1OGM7M4LTRX5Z3L4K0/__<u></u>journal.jif<div class="im"><br>
<br>
� � � � Any insight would be very valuable for our planning processes.<br>
<br>
� � � � Thanks,<br>
<br>
� � � � Matt<br>
<br>
<br></div>
� � � � ______________________________<u></u>___________________<br>
� � � � rabbitmq-discuss mailing list<br>
� � � � rabbitmq-discuss@lists.__<a href="http://rabbitmq.com" target="_blank">rabbi<u></u>tmq.com</a><br>
� � � � <mailto:<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@<u></u>lists.rabbitmq.com</a>><br>
� � � � <a href="https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/__<u></u>cgi-bin/mailman/listinfo/__<u></u>rabbitmq-discuss</a><br>
� � � � <<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a>><br>
<br>
<br>
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
</font></span></blockquote></div><br></div>