[rabbitmq-discuss] Throttling when Fast Producer, Slow Consumer

Wed Mar 14 14:59:17 GMT 2012

Hi Paul:

1. Yes, the spill-over happens regardless of your durability specifications.
Think of it as analogous to an OS with virtual memory and swapping, writing
pages to disk when physical RAM becomes precious:  You didn't ask for that
data to end up on disk, but it's being moved there temporarily due to need.
Note that this spilling of memory to disk is a RabbitMQ broker mechanism, not
an operating system one.

2. It's automatically handled by Rabbit, in response to the broker's memory
usage going above a configured fraction of the total memory available to the
Erlang VM.  That fraction, the "memory high watermark," can be adjusted in
your RabbitMQ configuration but generally one should trust the default.  If
it's set too low, messages may swap out to disk prematurely, and if it's set
too high, the mechanism may not engage until the broker is in significant
memory jeopardy.

Best regards,
Jerry

----- Original Message -----
From: "Paul M. Bell" <pbell at syncsort.com>
To: "Emile Joubert" <emile at rabbitmq.com>
Cc: "Jerry Kuch" <jerryk at vmware.com>, "Simone Busoli" <simone.busoli at gmail.com>, zmstone at gmail.com, "davorin rusevljan" <davorin.rusevljan at gmail.com>, rabbitmq-discuss at lists.rabbitmq.com
Sent: Wednesday, March 14, 2012 6:38:41 AM
Subject: RE: [rabbitmq-discuss] Throttling when Fast Producer, Slow Consumer

Emile,

Thanks for the x-message-ttl info.

A couple of questions about your prior e-mail where you said "queues may get very long and need to spill over to disk..."

1. Am I right to think that this "spill over" happens for non-durable exchanges and queues?
2. Is there a way to govern when that spill over occurs, e.g., when queue gets to a certain size, or is this auto-magically handled by Rabbit?

-Paul

-----Original Message-----
From: Emile Joubert [mailto:emile at rabbitmq.com]
Sent: Wednesday, March 14, 2012 9:19 AM
To: Bell, Paul M.
Cc: Jerry Kuch; Simone Busoli; zmstone at gmail.com; davorin.rusevljan at gmail.com; rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Throttling when Fast Producer, Slow Consumer

Hi Paul,

On 14/03/12 12:59, Bell, Paul M. wrote:
> Also, is "x-message-ttl" accessible
> by means of Spring AMQP?

Yes, these are simply queue arguments. Spring AMQP allows you to declare
queues with arguments.

> On a related matter, please consider the case where the consumers
> simply aren't there (i.e., it's not that they're slow, they're simply
> NOT). Yet the long-running producers continue to publish to the
> exchange. Assume that we're dealing with a non-durable topic
> exchange. Am I right in thinking that, absent any bindings, the
> broker will simply discard incoming messages?

Yes, by default such messages will be discarded.

You can have such messages sent to an alternative exchange by using the
"alternative exchange" extension:
http://www.rabbitmq.com/extensions.html#alternate-exchange

Also see the "mandatory" publish argument:
http://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.publish.mandatory
Publishers can have messages returned if they could not be routed.

-Emile

ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other  confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.