[rabbitmq-discuss] Stop producer and queue continue growing...

[Matthew Sackman]

On Fri, Mar 12, 2010 at 11:36:31AM -0300, Gustavo Aquino wrote:
> This is my biggest concern.

It's the nature of AMQP.

> Look about this scenario I have one Exchange and one persistent queue.I'm
> posting transient messages to this Exchange (to get better performance) if
> my Exchanges messages are in buffer and server crash I will be loose all
> messages in buffer and it is a big problem.... I cannot lose any messages,

Not true. There is no messaging product available (nor will there ever
be) that can guarantee it can't lose messages. Do you have infinite disk
space? Is there no event that can cause power loss? Even in the event of
the sun exploding you want to ensure no message loss?

The fact is, message loss is unavoidable. You can take many steps to
avoid this, by using transactions and buffering messages in the client
so that if the connection to the broker dies you can reconnect and
republish, but at that point you risk duplicates, so you need
application logic on the consumer side to watch for duplicates and
remove them as necessary.

> this messages are business it represent money, so if I loose one we loose
> money and It cannot happen, in other way I need to be the fastest we
> need overrides in nanoseconds or 1 millisecond is the limit, so post
> messages in persistent queue is out of the scope, or  have one way to be
> fast with persistent ?

Right, so you can't afford the delay of writing to disk, and yet you
want to ensure messages can't be lost in the event of a crash, or
comet strike. You may wish then to explore multiple brokers, publish, in
transactions to duplicate resources (queues) on every broker, and then
simultaneously consume from all of them, deduplicating as you go. Thus
you don't need to write to disk, and by shear strength of numbers, you
should be able to survive crashes. However, really these various brokers
should be in different data centres, probably on different continents,
so the 1ms max delay may be challenging...

Matthew