<div dir="ltr">Yep glad for the clarification and that's the behavior I'd actually want thinking about this. :) It's like a file load - all IO, no CPU. SO we use consumers to do the CPU processing - which is inherently slower in a ETL concept. We're basically using RabbitMQ for distributed ETL work (go distributed computing). Since file loads are intermittent, you wouldn't want to slow down file loads.<div>
<br></div><div>Thanks for the clarification and that makes it MUCH more clear!</div><div>Jason</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 10, 2014 at 10:41 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">On 10/07/2014 4:23PM, Jason McIntosh wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Interesting on the 3.3 change. SO I gotta ask about this situation on<br>
that (though it sounds off hand that'll fix the behavior I've seen on<br>
3.2.4). If a consumer is consumer at 80 msgs/second, but incoming rates<br>
are able to publish at 5k a second and the server CPU's/disks/etc.<br>
aren't being touched, will rabbit then put back pressure on the<br>
publishing rate forcing it down to 72 msgs/second (~10% less than<br>
consumes)?<br>
</blockquote>
<br></div>
No, no, no!<br>
<br>
(And this is why writing documentation is hard! At least for me.)<br>
<br>
I somewhat oversimplified. What really happens is that the queue will choose to do work that relates to consumers rather than publishers as long as:<br>
<br>
1) The queue is running flat out; it never has any idle time.<br>
2) The queue actually has work to do that pertains to consuming.<br>
3) Messages are being consumed at less than 110% of the rate they're being published.<br>
<br>
In the case you describe above, 2) will be false 99% of the time, at which point the queue will accept publishes as fast as it can.<br>
<br>
In other words, when I said "if consumers and producers are both going as fast as they can" I meant if the limiting factor is the queue itself, not whatever work they're doing.<br>
<br>
I hope that is clearer this time round.<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'd assume then that'd propagate up to the publisher if<br>
there's a chain of rabbit servers all running 3.3 - e.g. the code<br>
publish to rabbit which shovels to the remote rabbit where the code<br>
consumer runs. In this case the code publisher would get back pressure<br>
that'd reduce it's publish rate to ~72msgs/second?<br>
</blockquote>
<br></div>
Which would be a disaster. It's meant to be a queuing system, after all :-)<br>
<br>
Cheers, Simon<div class="HOEnZb"><div class="h5"><br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, Pivotal<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Jason McIntosh<br><a href="https://github.com/jasonmcintosh/" target="_blank">https://github.com/jasonmcintosh/</a><br>573-424-7612</div>
</div>