Thanks for your reply - comments inline below<br><br><div class="gmail_quote">On 6 May 2010 13:34, Martin Sustrik <span dir="ltr"><<a href="mailto:sustrik@250bpm.com" target="_blank">sustrik@250bpm.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Rob,<br>
<br>
First of all I would like to define what's MQ and what's DB. It'll be helpful to get the discussion more focused IMO:<br>
<br>
With DB data are long-lived while queries are transient one-off events.<br>
<br></blockquote><div><br>I think this is a slight over-simplification. AMQP aims to provide store and forward messaging where "store" is an important part of that equation. The storage may be transitory or it may be relatively long lived (I've just been dealing with a situation where messages had been sitting in a queue for several weeks, quite legitimately). Any notion of "queuing" is implicitly going to rely on some sort of storage: the queue. Some use cases of Messaging (in particular low-latency data distribution) may want to avoid queuing and simply drop messages that cannot be consumed immediately. While this is not the use case AMQP is designed for, it can conceivably still be used in this way. not that it is not a requirement for every node to support every distribution mode - moreover <br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
With MQ data are transient one-off events while queries are stable long-lived entities.<br>
<br>
Terminology:<br>
<br>
Data = record (DB) = message (MQ)<br>
Query = SQL statement (DB) = subscription (MQ)<div><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
It's wider than ordering alone. Node with unordered messages is basically a<br>
database table.<br>
</blockquote>
<br>
A Node with unordered messages would be some sort of Bag I would<br>
think... Unlike a DB table there would be no natural notion of a<br>
primary key, or necessarily any notion of a way to select the messages<br>
(though I guess the node *may* support certain AMQP1-0 filter types).<br>
Again, there is no requirement on implementers to write such a Node<br>
type (unless Michael has secretly put that in the document he is<br>
currently writing ;-) )... All that the core spec is doing is saying<br>
is "it's perfectly valid to have nodes that give FIFO ordering, LIFO<br>
ordering, or any other (non-)ordering you like". This is actually<br>
removing a burden on implementers rather than adding one (although the<br>
requirement for the same sort of pseudo-FIFO that 0-x requires will<br>
probably also be a requirement for AMQP 1-0 *Brokers*).<br>
</blockquote>
<br></div>
Yes. Explicitly not defining how the messages are stored and leaving it to layer above is perfectly valid approach. However, it should be done more consistently, see below...<div><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
However, there are also other DB concepts in the spec. Say a concept of<br>
non-destructive link -- which boils down to a SELECT statement. Or, say,<br>
it seems implied that a message cannot silently disappear from a node<br>
same way as record cannot silently disappear from a DB table.<br>
</blockquote>
<br>
There is absolutely no intent to prevent a node from silently<br>
"disappearing" a message that it is holding. All that<br>
"non-destructive" means is that the act of sending the message to the<br>
receiver does not in itself cause the message to be "disappeared".<br>
The two major use cases this functionality is designed for are Queue<br>
"browsing" (i.e. the ability to look at what is in your queue without<br>
consuming the messages) and the ability to implement "Topics" as a<br>
single entity (rather than the AMQP 0-x way of having a queue per<br>
consumer). The Topic case is potentially more interesting as a Topic<br>
node would have a policy that says "when all connected consumers have<br>
moved beyond a message in the *ordered* Topic, then the message may be<br>
deleted". This obviously requires some notion of ordering at the<br>
(Topic) node. Having implemented this functionality elsewhere, it<br>
certainly doesn't require a database table like data-structure - all<br>
it requires is that you hold pointers into the queue.<br>
</blockquote>
<br></div>
Let me give you just one example of what problems we are facing here: Destructive vs. non-destructive semantics break "stability" of message dispatch algorithm. In other words, which messages you get is not fully determined by your subscription, rather it depends on ordering of subscriptions in the broker. If destructive subscription is matched first, non-destructive subscription won't get the message. If they are matched in reverse order, both get the message. You do expect such behaviour with DB system -- ordering of SQL statements effects the results -- however, you don't want it to happen with MQ system.<br>
<br></blockquote><div><br>If you use destructive links at all then obviously (and correctly) which messages are sent to which consumers will depend completely on the order in which the node offers the messages to the links. This is the epected behaviour for queues. For instance a shared work queue Q with consumers A, B and C ... when a message M enters the queue it is the choice of the node which of the A, B, or C it offers the message to - the other two will never see it. So I disagree that this is not expected behaviour in Messaging. What you seem to be describing above is "routing" where there is no storage capability.<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
My suggestion would be to stretch the abstraction of ignoring the internal node state/algorithms to fully cover the node behaviour:<br>
<br>
1. Node is a named endpoint for links and that's it.<br>
<br>
2. There's no concept of message being resident on a node.<br>
<br>
3. There's no message ordering defined. This follows from the above.<br>
<br>
4. There's no message state machine. This follows from 2.<br>
<br>
3. There are no destructive/non-destructive links, just "links".<br>
<br></blockquote><div><br>I believe the spec already separates out these concerns into separate layers in a way which I think generally matches your categorization. Each Book in the spec represents an independent layer (with the
exception of the first book which is there to provide a gentle
introduction to the concepts).<br><br>To the Transport Layer (Book III) links are between link endpoints - there is no concept of message residence, ordering, or a state machine. This is what you are describing above I think.<br>
<br>Book IV, the Messaging Layer defines a particular form for describing the link endpoints. This defines a scheme for identifying whether you wish to take the messages from the sender (acquire/destructive) or simply look at the messages with taking them (browse/non-destructive). This is also the layer where we define how to request durability of the message, allow for the assigning of priorities, etc. This behaviour should be common to more than just "brokers" but also available in other AMQP intermediaries respecting the same Messaging layer. <br>
<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Adopting the points above would make core specification much cleaner. The burden of defining storage/ordering/routing etc. is left to "broker behaviour" specification.<br><font color="#888888">
<br></font></blockquote><div><br>Broker Behaviour will define a concrete set of capabilities that
something calling itself an "AMQP Broker" MUST provide. In effect it
is documenting constraints on the implementer as to the minimum
features they must provide through the Messaging Layer.<br><br>Even within a
container which implements the Messaging Layer as described in Book IV,
not every node need support all the options - it is expected that each
node will support only certain messaging patterns. For example a node
that implements only "routing" and not storage of any kind (something a bit like an 0-x Exchange) would not
support mixing distribution modes I imagine - either it would only
allow non-destructive distribution (acting as a fanout- every message
goes to every consumer) or only allow destructive distribution (acting
as a switch - each message goes to the first/only consumer that can
take it... or is dropped if there is no such consumer). <br><br>As an
aside, the names destructive/non-destructive could probably be improved
upon 9as they don't really properly capture the implied meaning) - but
no-one has yet come up with terms that can be universally agreed upon
:-)<br><br>Hope this is making things a bit clearer <br><br>-- Rob<br>
</div></div><br>