[rabbitmq-discuss] RabbitMQ Producer Flow Control

[Aaron Blew]

Hi all,
I'm currently using RabbitMQ 2.8.1 as the backbone of my Logstash
architecture.  Currently I've got 5 RabbitMQ servers running shovels that
push messages onto the "mothership", where data is consumed by a Logstash
agent that feeds into various endpoints.

The issue is that under high load the Logstash consumer on the mothership
isn't fast enough at to keep up with the data coming in (generally above
4,000 messages/sec).  Normally this wouldn't be a problem since we've got
capacity to buffer things on the mothership, but RabbitMQ starts rate
limiting the shovels that are sending data in, which compounds the issue
since the nodes shoveling data don't have as much IO capacity as the
mothership does.  If I stop the slow consumer on the mothership, the
shovels play data in as fast as they can acknowledge it.

Ideally I'd be able to turn off producer flow control on the mothership
side so the shovels wouldn't be slowed down except in cases where I'm
approaching high watermarks or something along those lines.

Can anyone offer any suggestions on how I might be able to adjust the
behavior?

Thanks,
-Aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130201/919d9ce2/attachment.htm>