[rabbitmq-discuss] Rate limiting queues

[Tim Watson]

Hi

On 16 Jan 2013, at 00:36, Pedro Werneck wrote:

> I'm running RabbitMQ 3.0.1 with celery 3.0.11 for a stack of applications based on the Python framework Flask, with a demand for a dozen million messages a day.
> 
> The problem is, some queues have to be rate limited dynamically during the day. We had a lot of problems with rate limiting the workers, using the rate_limit provided by celery and other client side solutions, and decided to look for some limiting on the deliver rate by the broker itself. I searched a lot and couldn't find anything like that, which surprised me, nor a reason why something like that isn't implemented.

Rabbit *does* rate limiting with regards flow control, so if the broker is struggling (e.g., consumers cannot keep up with producers) then back pressure will be exerted on the producers. There's nothing in place right now to rate limit based on other factors, though it's possible such a feature might be useful. How would you envisage something like that working? 

> I found a plugin that implements throttling between the exchange and the queue, but I'm not sure how reliable it is. https://github.com/flopezluis/rabbitmq-throttling-exchange
> 

I would be cautious about using that. The exchange will be called in the reader process, and I'm pretty certain that performing a blocking receive doesn't make sense there.

> I'm not familiar with Erlang and the RabbitMQ internals, so we adopted an ugly hackish solution for now, which is to have an input queue, an output queue, and a background process using a token bucket algorithm and amqp to get the messages from the input queue and send them back to the exchange and the output queue with the given rate. It isn't pretty, but is working reliably and we know everything that's going on.
> 

That sounds reasonably sensible to me. I assume you've read http://dougbarth.github.com/2011/06/10/keeping-the-rabbit-on-a-leash.html ??

> Is there any reliable solution I missed? If not, is there any way to improve this solution we adopted?
> 

It might be more efficient to run the rate limiting process inside the broker using the direct Erlang client to avoid the overhead of serialising data and transmitting it over the network. That's probably a half-way house to just doing this in the broker though.

The broker does have an effective credit-based flow control mechanism already. It might be possible to add additional 'sensors' to the broker which could be used to trigger changes in available credit and cause the flow control mechanism to kick in, though I'm not sure whether this is something that would be considered desirable. It certainly *sounds* interesting, but I suspect it might fall into the category of 'one too many knobs to twiddle' though. Rabbit has been made deliberately minimal with regards knob twiddling.

The *other* thing you might consider is rate limiting at the network level. Decent commercially available load balancers that handle non HTTP traffic can do this for you, though I've not seen any AMQP-aware offerings and this might impact negatively on performance if you can't configure the traffic manager to handle frame sizes correctly.

Cheers,
T