[rabbitmq-discuss] Rate limiting queues

[Pedro Werneck]

I'm running RabbitMQ 3.0.1 with celery 3.0.11 for a stack of applications 
based on the Python framework Flask, with a demand for a dozen million 
messages a day.

The problem is, some queues have to be rate limited dynamically during the 
day. We had a lot of problems with rate limiting the workers, using the 
rate_limit provided by celery and other client side solutions, and decided 
to look for some limiting on the deliver rate by the broker itself. I 
searched a lot and couldn't find anything like that, which surprised me, 
nor a reason why something like that isn't implemented. I found a plugin 
that implements throttling between the exchange and the queue, but I'm not 
sure how reliable it is. 
https://github.com/flopezluis/rabbitmq-throttling-exchange

I'm not familiar with Erlang and the RabbitMQ internals, so we adopted an 
ugly hackish solution for now, which is to have an input queue, an output 
queue, and a background process using a token bucket algorithm and amqp to 
get the messages from the input queue and send them back to the exchange 
and the output queue with the given rate. It isn't pretty, but is working 
reliably and we know everything that's going on.

Is there any reliable solution I missed? If not, is there any way to 
improve this solution we adopted?


Thanks for any help


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130115/c571a610/attachment.htm>