[rabbitmq-discuss] Throttling when Fast Producer, Slow Consumer

[Bell, Paul M.]

Guys,

I fell asleep last night thinking about my reply to Jerry and Simone, only to find several others had weighed in on this issue.

In re flow control in general: Simone, yeah "flow control" is exactly the right way to characterize the problem. I am a bit familiar with the prefetch value when dealing with messages inbound to a consumer, but found no analog in the other direction.

In re use of the HTTP API: Jerry, I completely agree that such an approach, while tempting, really does violate the "information hiding" aspect of decoupled producers and consumers. That said, can you say more about how you understand the difference between "monitoring the system" and using the HTTP API to "keep a track on the queue depths?"

In re use of queue_declare: zmstone, thank you for this. Do you happen to know if this is available via Spring AMQP or is it only available via the Java or .NET AMQP clients?

In re use of credits: Davorin, can you say more about this? This is the first time I've heard of "credits" in the context of AMQP. (Hmm...now that I think of it, I'm not even sure which version of AMQP RabbitMQ is an implementation of).

In re mismatched rates of production/consumption: Emile, no this is not a permanent feature. I consider it a pathological and anomalous state. But it may happen. What is your recommended way for a producer who is writing to an exchange to monitor the length of a queue on "the other side" of the exchange? Also, is "x-message-ttl" accessible by means of Spring AMQP?

On a related matter, please consider the case where the consumers simply aren't there (i.e., it's not that they're slow, they're simply NOT). Yet the long-running producers continue to publish to the exchange. Assume that we're dealing with a non-durable topic exchange. Am I right in thinking that, absent any bindings, the broker will simply discard incoming messages?

Thank you all.

-Paul






-----Original Message-----
From: Jerry Kuch [mailto:jerryk at vmware.com] 
Sent: Tuesday, March 13, 2012 7:01 PM
To: Bell, Paul M.
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Throttling when Fast Producer, Slow Consumer

Hi, Paul:  You could use the management HTTP API to keep a track on
queue depths and then have your producer dynamically adjust its 
behavior if it notices that consumption isn't keeping up, but you
should really think about whether this is something you want to do.

It ends up building a lot of of hand holding awareness of the messaging
mechanics into your endpoint applications, which in some ways goes against
the spirit of using a messaging service in between your producers and
consumers in the first place.  You might be better off monitoring the
system, and spinning up additional consumers to drain especially busy 
queues than building the sort of logic you describe...

Best regards,
Jerry

----- Original Message -----
From: "Paul M. Bell" <pbell at syncsort.com>
To: rabbitmq-discuss at lists.rabbitmq.com
Sent: Tuesday, March 13, 2012 3:26:15 PM
Subject: [rabbitmq-discuss] Throttling when Fast Producer, Slow Consumer

Hi All,

What are the Rabbit "best practices" ways of handling a situation where the consumer is v-e-r-y slow and the producer is super-fast. IOW: I am trying to understand the best ways to throttle the producer in such a case. Assume that neither queue nor its topic exchange are durable.

Can producer obtain information about the state of the queue, e.g., the number of unACKed messages, its size in bytes or number of messages, etc? Or is another approach indicated?

Thanks very much.

-Paul




ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other  confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss