[rabbitmq-discuss] Controlling over-producers

Wed May 7 22:39:41 BST 2008

Edwin,

Edwin Fine wrote:
>  2. The consumer often gets flooded with messages from the producer
>  when it does some time-based operations. In Erlang, this means that
>  the consumer's Erlang message queue grows to a couple hundred
>  messages, which is not what I want. I want the producer to stop
>  sending messages when the consumer is too busy.

Do you require end-to-end flow control, i.e. you need to throttle the 
publisher of the messages when the consumer cannot keep up, or just flow 
control between the broker and the consumer?

>  Messages must not be lost and are marked persistent, so I want them to
>  stay on disk until the consumer acknowledges successful processing. Am
>  I correct to assume RabbitMQ does this if I have marked the queue as
>  persistent and have not replied to the basic.deliver with an ack?

The messages themselves need to be marked as persistent, and the queue 
needs to be marked as durable.

>  3. The channel.flow object is not currently implemented in RabbitMQ; I
>  think this would be perfect to do this.

channel.flow is rather imperfect :) For starters, it operates on an 
entire channel, whereas it looks like you want it to operate on a 
per-consumer basis.

>  4. IIRC, a suggestion was for the consumer to basic.cancel itself
>  until the load reduces, and then re-subscribe (basic.consume) once the
>  overflow condition has abated. This seems heavy-handed, and possibly
>  resource-intensive, especially since it might happen relatively
>  frequently.

The frequency is unlikely to be an issue - basic.consume/cancel are 
pretty efficient.

The more serious problem may be prefetching. By the time your consumer 
wants to start throttling there may already be a large number of 
messages in flight. The client will still receive these after issuing a 
basic.cancel, or, for that matter, channel.flow.

You can address that by switching to use basic.get, though that is 
non-blocking.

Matthias