[rabbitmq-discuss] Throttling when Fast Producer, Slow Consumer

Wed Mar 14 13:04:54 GMT 2012

In re to your last question, regardless of the exchange type and whether the exchange is durable or not messages with no destination are always discarded (although I think a dead letter mechanism is appearing soon)

-----Original Message-----
From: rabbitmq-discuss-bounces at lists.rabbitmq.com [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Bell, Paul M.
Sent: mercoledì 14 marzo 2012 13:59
To: Jerry Kuch; Simone Busoli; zmstone at gmail.com; davorin.rusevljan at gmail.com; Emile Joubert
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Throttling when Fast Producer, Slow Consumer

Guys,

I fell asleep last night thinking about my reply to Jerry and Simone, only to find several others had weighed in on this issue.

In re flow control in general: Simone, yeah "flow control" is exactly the right way to characterize the problem. I am a bit familiar with the prefetch value when dealing with messages inbound to a consumer, but found no analog in the other direction.

In re use of the HTTP API: Jerry, I completely agree that such an approach, while tempting, really does violate the "information hiding" aspect of decoupled producers and consumers. That said, can you say more about how you understand the difference between "monitoring the system" and using the HTTP API to "keep a track on the queue depths?"

In re use of queue_declare: zmstone, thank you for this. Do you happen to know if this is available via Spring AMQP or is it only available via the Java or .NET AMQP clients?

In re use of credits: Davorin, can you say more about this? This is the first time I've heard of "credits" in the context of AMQP. (Hmm...now that I think of it, I'm not even sure which version of AMQP RabbitMQ is an implementation of).

In re mismatched rates of production/consumption: Emile, no this is not a permanent feature. I consider it a pathological and anomalous state. But it may happen. What is your recommended way for a producer who is writing to an exchange to monitor the length of a queue on "the other side" of the exchange? Also, is "x-message-ttl" accessible by means of Spring AMQP?

On a related matter, please consider the case where the consumers simply aren't there (i.e., it's not that they're slow, they're simply NOT). Yet the long-running producers continue to publish to the exchange. Assume that we're dealing with a non-durable topic exchange. Am I right in thinking that, absent any bindings, the broker will simply discard incoming messages?

Thank you all.

-Paul

-----Original Message-----
From: Jerry Kuch [mailto:jerryk at vmware.com]
Sent: Tuesday, March 13, 2012 7:01 PM
To: Bell, Paul M.
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Throttling when Fast Producer, Slow Consumer

Hi, Paul:  You could use the management HTTP API to keep a track on queue depths and then have your producer dynamically adjust its behavior if it notices that consumption isn't keeping up, but you should really think about whether this is something you want to do.

It ends up building a lot of of hand holding awareness of the messaging mechanics into your endpoint applications, which in some ways goes against the spirit of using a messaging service in between your producers and consumers in the first place.  You might be better off monitoring the system, and spinning up additional consumers to drain especially busy queues than building the sort of logic you describe...

Best regards,
Jerry

----- Original Message -----
From: "Paul M. Bell" <pbell at syncsort.com>
To: rabbitmq-discuss at lists.rabbitmq.com
Sent: Tuesday, March 13, 2012 3:26:15 PM
Subject: [rabbitmq-discuss] Throttling when Fast Producer, Slow Consumer

Hi All,

What are the Rabbit "best practices" ways of handling a situation where the consumer is v-e-r-y slow and the producer is super-fast. IOW: I am trying to understand the best ways to throttle the producer in such a case. Assume that neither queue nor its topic exchange are durable.

Can producer obtain information about the state of the queue, e.g., the number of unACKed messages, its size in bytes or number of messages, etc? Or is another approach indicated?

Thanks very much.

-Paul

ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other  confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_________________________________________________________________________________________________________________
Questo messaggio è da intendersi esclusivamente ad uso del destinatario e può contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non è il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione è 
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from 
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the 
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from 
your system.
_________________________________________________________________________________________________________________