<div dir="ltr">Hi all,<div>I'm currently using RabbitMQ 2.8.1 as the backbone of my Logstash architecture. Currently I've got 5 RabbitMQ servers running shovels that push messages onto the "mothership", where data is consumed by a Logstash agent that feeds into various endpoints.</div>
<div><br></div><div style>The issue is that under high load the Logstash consumer on the mothership isn't fast enough at to keep up with the data coming in (generally above 4,000 messages/sec). Normally this wouldn't be a problem since we've got capacity to buffer things on the mothership, but RabbitMQ starts rate limiting the shovels that are sending data in, which compounds the issue since the nodes shoveling data don't have as much IO capacity as the mothership does. If I stop the slow consumer on the mothership, the shovels play data in as fast as they can acknowledge it.</div>
<div style><br></div><div style>Ideally I'd be able to turn off producer flow control on the mothership side so the shovels wouldn't be slowed down except in cases where I'm approaching high watermarks or something along those lines. </div>
<div style><br></div><div style>Can anyone offer any suggestions on how I might be able to adjust the behavior?</div><div style><br></div><div style>Thanks,</div><div style>-Aaron</div></div>