<div dir="ltr">Hey Simon,<div>Thanks for getting back to me. After upgrading to 2.8.7 I'm still unable to flow messages in to the head node faster than 4,000/sec while the consumer is working. I appreciate that flow control is trying to prevent the server from being over-run, but in some situations it would be very beneficial for an administrator to override this behavior. Are there any plans to add this kind of behavior? I'm currently seriously considering a switch to ActiveMQ/stomp since it gives me the knobs to tweak this behavior both in real time and via configuration.</div>
<div><br></div><div style>Thanks!</div><div style>-Aaron</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Feb 4, 2013 at 3:30 AM, Simon MacMullen <span dir="ltr"><<a href="mailto:simon@rabbitmq.com" target="_blank">simon@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi. There's a bug in versions of RabbitMQ prior to 2.8.4 where under some circumstances queues can over-prioritise getting rid of message to such an extent that they don't accept any more until empty.<br>
<br>
I recommend an upgrade to 2.8.7 at least, or preferably 3.0.2.<br>
<br>
And in this case flow control is just the messenger, so you don't want to shoot it :-)<br>
<br>
Cheers, Simon<div><div class="h5"><br>
<br>
On 01/02/13 17:31, Aaron Blew wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hi all,<br>
I'm currently using RabbitMQ 2.8.1 as the backbone of my Logstash<br>
architecture. Currently I've got 5 RabbitMQ servers running shovels<br>
that push messages onto the "mothership", where data is consumed by a<br>
Logstash agent that feeds into various endpoints.<br>
<br>
The issue is that under high load the Logstash consumer on the<br>
mothership isn't fast enough at to keep up with the data coming in<br>
(generally above 4,000 messages/sec). Normally this wouldn't be a<br>
problem since we've got capacity to buffer things on the mothership, but<br>
RabbitMQ starts rate limiting the shovels that are sending data in,<br>
which compounds the issue since the nodes shoveling data don't have as<br>
much IO capacity as the mothership does. If I stop the slow consumer on<br>
the mothership, the shovels play data in as fast as they can acknowledge it.<br>
<br>
Ideally I'd be able to turn off producer flow control on the mothership<br>
side so the shovels wouldn't be slowed down except in cases where I'm<br>
approaching high watermarks or something along those lines.<br>
<br>
Can anyone offer any suggestions on how I might be able to adjust the<br>
behavior?<br>
<br>
Thanks,<br>
-Aaron<br>
<br>
<br></div></div><div class="im">
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
<br>
</div></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
-- <br>
Simon MacMullen<br>
RabbitMQ, VMware<br>
</font></span></blockquote></div><br></div>