<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">(Replying again but using Reply-All to ensure rabbitmq-discuss forum sees my response)</span><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Hi Simon,</span><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Thanks a lot for your response. Okay, I just wanted to make sure I didn't have something misconfigured. If the throughput I'm seeing is considered "normal" given the type of machines I'm running on, then that is a huge help to me. I had been wondering if those numbers were considered good, bad, etc. Thanks!</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px"><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Apr 25, 2013 at 6:59 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Karl. I suspect you are not really seeing a bottleneck with acknowledgements, but rather an optimisation in autoack mode. When you publish a persistent message to an empty queue with a non-blocked autoack consumer RabbitMQ will not persist the message to disc - there's no point. The message can go straight to the consumer, and then it's gone; it can never be requeued.<br>
<br>
So I suspect that's the difference you're seeing. And I'm afraid 5-8k msg/s is roughly what I would expect for persistent messages on a reasonable machine.<br>
<br>
Cheers, Simon<br>
<br>
On 24/04/13 17:26, Karl Rieb wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
I am trying to improve the message throughput for a RabbitMQ queue in an<br>
Amazon cloud instance and am noticing a *significant* drop in<br>
performance when enabling acknowledgements for consumer of a durable<br>
queue (with persisted messages). The real problem is that the<br>
bottleneck appears to be on the rabbit node and not with the consumers,<br>
so adding more consumers does not improve the throughput (or help drain<br>
the queue any quicker). As a matter of fact, adding new consumers will<br>
just slow down existing consumers so everyone ends up consuming at a<br>
slower rate, preventing overall throughput from changing.<br>
<br>
Trying to do batch acknowledgements using the Multiple flag helps a bit<br>
(8k msgs/s vs 5.5k msgs/s) but not much compared to the initial drop.<br>
It is only when I turn on *auto_ack* for the consumers that I see the<br>
performance shoot *way *back up and when I start seeing a linear<br>
increase in throughput as I add more consumers.<br>
<br>
Is this expected behavior? Is there a way to configure the rabbit node<br>
so it doesn't hit this bottleneck with acknowledgements?<br>
<br>
Here is the sample code I'm using to test the throughput:<br>
<br>
Publisher:<br>
<br>
#!/usr/bin/python<br>
<br>
import pika<br>
<br>
creds = pika.PlainCredentials('guest',<u></u>'guest')<br>
conn =<br>
pika.BlockingConnection(pika.<u></u>ConnectionParameters(host='10.<u></u>10.1.123', credentials=creds))<br>
chan = conn.channel()<br>
<br>
while True:<br>
chan.basic_publish(exchange='<u></u>simple_exchange',<br>
routing_key='simple_queue', body='',<br>
properties=pika.<u></u>BasicProperties(delivery_mode=<u></u>2))<br>
<br>
<br>
Consumer:<br>
<br>
#!/usr/bin/python<br>
<br>
import pika<br>
<br>
def callback(chan, method, properties, body):<br>
chan.basic_ack(delivery_tag=<u></u>method.delivery_tag, multiple=False)<br>
<br>
creds = pika.PlainCredentials('guest',<u></u>'guest')<br>
conn =<br>
pika.BlockingConnection(pika.<u></u>ConnectionParameters(host='10.<u></u>10.1.123', credentials=creds))<br>
chan = conn.channel()<br>
<br>
chan.basic_consume(callback, queue='simple_queue', no_ack=False)<br>
chan.basic_qos(prefetch_count=<u></u>1000)<br>
chan.start_consuming()<br>
<br>
<br>
I spawn multiple processes for the producers and multiple for the<br>
consumer (so there is no python interpreter locking issues since each<br>
runs in its own interpreter instance). I'm using an an Amazon<br>
*c1.xlarge *(8 virtual cores and "high" IO) Ubuntu 12.04 LTS instance<br>
with RabbitMQ version 3.0.4-1 and an Amazon ephemeral disk (in<br>
production we would use an EBS volume instead). The queue is marked<br>
*Durable* and my messages all use *delivery_mode* 2 (persist).<br>
<br>
Below are the performance numbers. For each test I use 2 publishers<br>
processes and 6 consumer processes (where 3 different machines host 2<br>
consumers each). The producers and consumers are all on *separate*<br>
machines from the rabbit node. Throughput measurements were done using<br>
the RabbitMQ management UI and linux utility top. Python was compiled<br>
to pyc files before running.<br>
<br>
*no_ack = True:*<br>
rate = 24,000/s<br>
single consumer CPU = 65%<br>
single publisher CPU = 80% (flow control enabled and being enforced)<br>
(beam.smp) rabbit CPU = 400% (of 800%, 8 cores) -> 0.0%wa 11.5%sy<br>
<br>
*no_ack = False (manual acks per message):*<br>
rate = 5,500/s<br>
single consumer CPU = 20%<br>
single publisher CPU = 20% (flow control enabled and being enforced)<br>
(beam.smp) rabbit CPU = 300% (of 800%, 8 cores) -> 4.5%wa 10.0%sy<br>
The most notable difference besides the throughput are the I/O waits<br>
when ACKs are enabled (4.5% vs 0.0%). This leads me to believe that the<br>
rabbit node is being bottlenecked by performing I/O operations for ACK<br>
bookkeeping. The I/O doesn't appear to be a problem for persisting the<br>
published messages since I'm *guessing* that rabbit is buffering those<br>
and syncing them to disk in batches. Does this mean the<br>
acknowledgements are not also being buffered before synced with disk?<br>
Can I configure the rabbit node to change this behavior to help speed<br>
up the acknowledgements? I'm not using transactions in the example<br>
code above, so I don't need any strict guarantees that ACKs were written<br>
to disk before returning.<br>
<br>
Thanks,<br>
Karl<br>
<br>
P.S. I wrote the same sample consumer code in Ruby to see if there was a<br>
difference (in case there was a Python issue), but the numbers were<br>
about the same.<br>
<br>
<br>
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
</font></span></blockquote></div><br></div>