[rabbitmq-discuss] Controlling over-producers
matthias at lshift.net
Wed May 7 22:39:41 BST 2008
Edwin Fine wrote:
> 2. The consumer often gets flooded with messages from the producer
> when it does some time-based operations. In Erlang, this means that
> the consumer's Erlang message queue grows to a couple hundred
> messages, which is not what I want. I want the producer to stop
> sending messages when the consumer is too busy.
Do you require end-to-end flow control, i.e. you need to throttle the
publisher of the messages when the consumer cannot keep up, or just flow
control between the broker and the consumer?
> Messages must not be lost and are marked persistent, so I want them to
> stay on disk until the consumer acknowledges successful processing. Am
> I correct to assume RabbitMQ does this if I have marked the queue as
> persistent and have not replied to the basic.deliver with an ack?
The messages themselves need to be marked as persistent, and the queue
needs to be marked as durable.
> 3. The channel.flow object is not currently implemented in RabbitMQ; I
> think this would be perfect to do this.
channel.flow is rather imperfect :) For starters, it operates on an
entire channel, whereas it looks like you want it to operate on a
> 4. IIRC, a suggestion was for the consumer to basic.cancel itself
> until the load reduces, and then re-subscribe (basic.consume) once the
> overflow condition has abated. This seems heavy-handed, and possibly
> resource-intensive, especially since it might happen relatively
The frequency is unlikely to be an issue - basic.consume/cancel are
The more serious problem may be prefetching. By the time your consumer
wants to start throttling there may already be a large number of
messages in flight. The client will still receive these after issuing a
basic.cancel, or, for that matter, channel.flow.
You can address that by switching to use basic.get, though that is
More information about the rabbitmq-discuss