[rabbitmq-discuss] Stop producer and queue continue growing...

[Gustavo Aquino]

Matthew,


On Fri, Mar 12, 2010 at 11:58 AM, Matthew Sackman <matthew at lshift.net>wrote:

> On Fri, Mar 12, 2010 at 11:36:31AM -0300, Gustavo Aquino wrote:
> > This is my biggest concern.
>
> It's the nature of AMQP.
>
> > Look about this scenario I have one Exchange and one persistent queue.I'm
> > posting transient messages to this Exchange (to get better performance)
> if
> > my Exchanges messages are in buffer and server crash I will be loose all
> > messages in buffer and it is a big problem.... I cannot lose any
> messages,
>
> Not true. There is no messaging product available (nor will there ever
> be) that can guarantee it can't lose messages. Do you have infinite disk
> space? Is there no event that can cause power loss? Even in the event of
> the sun exploding you want to ensure no message loss?
>

If sun exploding money will be the minor problem of us. ;-)

We don't have infinite disk space, but the primary proposal is monitoring to
be able to be pro active and do something before problem happen.

Look,for example I'm monitoring queue size, and look that we are coming to
our limit, one way to guarantee the messages inside server is stopping
producers and redirect producers to other server, so in theory I can
guarantee that messages inside server1 will be consumed and server not loose
messages or crash.


Or how to guarantee that one message posted will be consumed ? consume don't
know about the message posted.


>
> The fact is, message loss is unavoidable. You can take many steps to
> avoid this, by using transactions and buffering messages in the client
> so that if the connection to the broker dies you can reconnect and
> republish, but at that point you risk duplicates, so you need
> application logic on the consumer side to watch for duplicates and
> remove them as necessary.
>
> > this messages are business it represent money, so if I loose one we loose
> > money and It cannot happen, in other way I need to be the fastest we
> > need overrides in nanoseconds or 1 millisecond is the limit, so post
> > messages in persistent queue is out of the scope, or  have one way to be
> > fast with persistent ?
>
> Right, so you can't afford the delay of writing to disk, and yet you
> want to ensure messages can't be lost in the event of a crash, or
> comet strike. You may wish then to explore multiple brokers, publish, in
> transactions to duplicate resources (queues) on every broker, and then
> simultaneously consume from all of them, deduplicating as you go. Thus
> you don't need to write to disk, and by shear strength of numbers, you
> should be able to survive crashes. However, really these various brokers
> should be in different data centres, probably on different continents,
> so the 1ms max delay may be challenging...
>

How to duplicate resources inside multiples brokers using Rabbitmq ? I saw
that Rabbit don't have a default way to do a Cluster HA, your based proposal
is HP.


>
> Matthew
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100312/91a1092b/attachment.htm