[rabbitmq-discuss] request for help!

Thu May 6 13:51:08 BST 2010

Thanks for your reply - comments inline below

On 6 May 2010 13:34, Martin Sustrik <sustrik at 250bpm.com> wrote:

> Hi Rob,
>
> First of all I would like to define what's MQ and what's DB. It'll be
> helpful to get the discussion more focused IMO:
>
> With DB data are long-lived while queries are transient one-off events.
>
>
I think this is a slight over-simplification.  AMQP aims to provide store
and forward messaging where "store" is an important part of that equation.
The storage may be transitory or it may be relatively long lived (I've just
been dealing with a situation where messages had been sitting in a queue for
several weeks, quite legitimately).  Any notion of "queuing" is implicitly
going to rely on some sort of storage: the queue.  Some use cases of
Messaging (in particular low-latency data distribution) may want to avoid
queuing and simply drop messages that cannot be consumed immediately.  While
this is not the use case AMQP is designed for, it can conceivably still be
used in this way.  not that it is not a requirement for every node to
support every distribution mode - moreover

> With MQ data are transient one-off events while queries are stable
> long-lived entities.
>
> Terminology:
>
> Data = record (DB) = message (MQ)
> Query = SQL statement (DB) = subscription (MQ)
>
>
>  It's wider than ordering alone. Node with unordered messages is basically
>>> a
>>> database table.
>>>
>>
>> A Node with unordered messages would be some sort of Bag I would
>> think... Unlike a DB table there would be no natural notion of a
>> primary key, or necessarily any notion of a way to select the messages
>> (though I guess the node *may* support certain AMQP1-0 filter types).
>> Again, there is no requirement on implementers to write such a Node
>> type (unless Michael has secretly put that in the document he is
>> currently writing ;-) )... All that the core spec is doing is saying
>> is "it's perfectly valid to have nodes that give FIFO ordering, LIFO
>> ordering, or any other (non-)ordering you like".  This is actually
>> removing a burden on implementers rather than adding one (although the
>> requirement for the same sort of pseudo-FIFO that 0-x requires will
>> probably also be a requirement for AMQP 1-0 *Brokers*).
>>
>
> Yes. Explicitly not defining how the messages are stored and leaving it to
> layer above is perfectly valid approach. However, it should be done more
> consistently, see below...
>
>
>  However, there are also other DB concepts in the spec. Say a concept of
>>> non-destructive link -- which boils down to a SELECT statement. Or, say,
>>> it seems implied that a message cannot silently disappear from a node
>>> same way as record cannot silently disappear from a DB table.
>>>
>>
>> There is absolutely no intent to prevent a node from silently
>> "disappearing" a message that it is holding.  All that
>> "non-destructive" means is that the act of sending the message to the
>> receiver does not in itself cause the message to be "disappeared".
>> The two major use cases this functionality is designed for are Queue
>> "browsing" (i.e. the ability to look at what is in your queue without
>> consuming the messages) and the ability to implement "Topics" as a
>> single entity (rather than the AMQP 0-x way of having a queue per
>> consumer).  The Topic case is potentially more interesting as a Topic
>> node would have a policy that says "when all connected consumers have
>> moved beyond a message in the *ordered* Topic, then the message may be
>> deleted".  This obviously requires some notion of ordering at the
>> (Topic) node.  Having implemented this functionality elsewhere, it
>> certainly doesn't require a database table like data-structure - all
>> it requires is that you hold pointers into the queue.
>>
>
> Let me give you just one example of what problems we are facing here:
> Destructive vs. non-destructive semantics break "stability" of message
> dispatch algorithm. In other words, which messages you get is not fully
> determined by your subscription, rather it depends on ordering of
> subscriptions in the broker. If destructive subscription is matched first,
> non-destructive subscription won't get the message. If they are matched in
> reverse order, both get the message. You do expect such behaviour with DB
> system -- ordering of SQL statements effects the results -- however, you
> don't want it to happen with MQ system.
>
>
If you use destructive links at all then obviously (and correctly) which
messages are sent to which consumers will depend completely on the order in
which the node offers the messages to the links.  This is the epected
behaviour for queues.  For instance a shared work queue Q with consumers A,
B and C ... when a message M enters the queue it is the choice of the node
which of the A, B, or C it offers the message to - the other two will never
see it.  So I disagree that this is not expected behaviour in Messaging.
What you seem to be describing above is "routing" where there is no storage
capability.

>
> My suggestion would be to stretch the abstraction of ignoring the internal
> node state/algorithms to fully cover the node behaviour:
>
> 1. Node is a named endpoint for links and that's it.
>
> 2. There's no concept of message being resident on a node.
>
> 3. There's no message ordering defined. This follows from the above.
>
> 4. There's no message state machine. This follows from 2.
>
> 3. There are no destructive/non-destructive links, just "links".
>
>
I believe the spec already separates out these concerns into separate layers
in a way which I think generally matches your categorization.  Each Book in
the spec represents an independent layer (with the exception of the first
book which is there to provide a gentle introduction to the concepts).

To the Transport Layer (Book III) links are between link endpoints - there
is no concept of message residence, ordering, or a state machine.  This is
what you are describing above I think.

Book IV, the Messaging Layer defines a particular form for describing the
link endpoints.  This defines a scheme for identifying whether you wish to
take the messages from the sender (acquire/destructive) or simply look at
the messages with taking them (browse/non-destructive).  This is also the
layer where we define how to request durability of the message, allow for
the assigning of priorities, etc.  This behaviour should be common to more
than just "brokers" but also available in other AMQP intermediaries
respecting the same Messaging layer.

> Adopting the points above would make core specification much cleaner. The
> burden of defining storage/ordering/routing etc. is left to "broker
> behaviour" specification.
>
>
Broker Behaviour will define a concrete set of capabilities that something
calling itself an "AMQP Broker" MUST provide.  In effect it is documenting
constraints on the implementer as to the minimum features they must provide
through the Messaging Layer.

Even within a container which implements the Messaging Layer as described in
Book IV, not every node need support all the options - it is expected that
each node will support only certain messaging patterns.  For example a node
that implements only "routing" and not storage of any kind (something a bit
like an 0-x Exchange) would not support mixing distribution modes I imagine
- either it would only allow non-destructive distribution (acting as a
fanout- every message goes to every consumer) or only allow destructive
distribution (acting as a switch - each message goes to the first/only
consumer that can take it... or is dropped if there is no such consumer).

As an aside, the names destructive/non-destructive could probably be
improved upon 9as they don't really properly capture the implied meaning) -
but no-one has yet come up with terms that can be universally agreed upon
:-)

Hope this is making things a bit clearer

-- Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100506/f4f5614d/attachment-0001.htm 