[rabbitmq-discuss] Rate limiting queues
pjwerneck at gmail.com
Wed Jan 16 00:36:50 GMT 2013
I'm running RabbitMQ 3.0.1 with celery 3.0.11 for a stack of applications
based on the Python framework Flask, with a demand for a dozen million
messages a day.
The problem is, some queues have to be rate limited dynamically during the
day. We had a lot of problems with rate limiting the workers, using the
rate_limit provided by celery and other client side solutions, and decided
to look for some limiting on the deliver rate by the broker itself. I
searched a lot and couldn't find anything like that, which surprised me,
nor a reason why something like that isn't implemented. I found a plugin
that implements throttling between the exchange and the queue, but I'm not
sure how reliable it is.
I'm not familiar with Erlang and the RabbitMQ internals, so we adopted an
ugly hackish solution for now, which is to have an input queue, an output
queue, and a background process using a token bucket algorithm and amqp to
get the messages from the input queue and send them back to the exchange
and the output queue with the given rate. It isn't pretty, but is working
reliably and we know everything that's going on.
Is there any reliable solution I missed? If not, is there any way to
improve this solution we adopted?
Thanks for any help
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss