[rabbitmq-discuss] RabbitMQ Producer Flow Control

[Simon MacMullen]

Hi. There's a bug in versions of RabbitMQ prior to 2.8.4 where under 
some circumstances queues can over-prioritise getting rid of message to 
such an extent that they don't accept any more until empty.

I recommend an upgrade to 2.8.7 at least, or preferably 3.0.2.

And in this case flow control is just the messenger, so you don't want 
to shoot it :-)

Cheers, Simon

On 01/02/13 17:31, Aaron Blew wrote:
> Hi all,
> I'm currently using RabbitMQ 2.8.1 as the backbone of my Logstash
> architecture.  Currently I've got 5 RabbitMQ servers running shovels
> that push messages onto the "mothership", where data is consumed by a
> Logstash agent that feeds into various endpoints.
>
> The issue is that under high load the Logstash consumer on the
> mothership isn't fast enough at to keep up with the data coming in
> (generally above 4,000 messages/sec).  Normally this wouldn't be a
> problem since we've got capacity to buffer things on the mothership, but
> RabbitMQ starts rate limiting the shovels that are sending data in,
> which compounds the issue since the nodes shoveling data don't have as
> much IO capacity as the mothership does.  If I stop the slow consumer on
> the mothership, the shovels play data in as fast as they can acknowledge it.
>
> Ideally I'd be able to turn off producer flow control on the mothership
> side so the shovels wouldn't be slowed down except in cases where I'm
> approaching high watermarks or something along those lines.
>
> Can anyone offer any suggestions on how I might be able to adjust the
> behavior?
>
> Thanks,
> -Aaron
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>


-- 
Simon MacMullen
RabbitMQ, VMware