[rabbitmq-discuss] request for help!

Thu May 6 12:34:27 BST 2010

Hi Rob,

First of all I would like to define what's MQ and what's DB. It'll be 
helpful to get the discussion more focused IMO:

With DB data are long-lived while queries are transient one-off events.

With MQ data are transient one-off events while queries are stable 
long-lived entities.

Terminology:

Data = record (DB) = message (MQ)
Query = SQL statement (DB) = subscription (MQ)

>> It's wider than ordering alone. Node with unordered messages is basically a
>> database table.
> 
> A Node with unordered messages would be some sort of Bag I would
> think... Unlike a DB table there would be no natural notion of a
> primary key, or necessarily any notion of a way to select the messages
> (though I guess the node *may* support certain AMQP1-0 filter types).
> Again, there is no requirement on implementers to write such a Node
> type (unless Michael has secretly put that in the document he is
> currently writing ;-) )... All that the core spec is doing is saying
> is "it's perfectly valid to have nodes that give FIFO ordering, LIFO
> ordering, or any other (non-)ordering you like".  This is actually
> removing a burden on implementers rather than adding one (although the
> requirement for the same sort of pseudo-FIFO that 0-x requires will
> probably also be a requirement for AMQP 1-0 *Brokers*).

Yes. Explicitly not defining how the messages are stored and leaving it 
to layer above is perfectly valid approach. However, it should be done 
more consistently, see below...

>> However, there are also other DB concepts in the spec. Say a concept of
>> non-destructive link -- which boils down to a SELECT statement. Or, say,
>> it seems implied that a message cannot silently disappear from a node
>> same way as record cannot silently disappear from a DB table.
> 
> There is absolutely no intent to prevent a node from silently
> "disappearing" a message that it is holding.  All that
> "non-destructive" means is that the act of sending the message to the
> receiver does not in itself cause the message to be "disappeared".
> The two major use cases this functionality is designed for are Queue
> "browsing" (i.e. the ability to look at what is in your queue without
> consuming the messages) and the ability to implement "Topics" as a
> single entity (rather than the AMQP 0-x way of having a queue per
> consumer).  The Topic case is potentially more interesting as a Topic
> node would have a policy that says "when all connected consumers have
> moved beyond a message in the *ordered* Topic, then the message may be
> deleted".  This obviously requires some notion of ordering at the
> (Topic) node.  Having implemented this functionality elsewhere, it
> certainly doesn't require a database table like data-structure - all
> it requires is that you hold pointers into the queue.

Let me give you just one example of what problems we are facing here: 
Destructive vs. non-destructive semantics break "stability" of message 
dispatch algorithm. In other words, which messages you get is not fully 
determined by your subscription, rather it depends on ordering of 
subscriptions in the broker. If destructive subscription is matched 
first, non-destructive subscription won't get the message. If they are 
matched in reverse order, both get the message. You do expect such 
behaviour with DB system -- ordering of SQL statements effects the 
results -- however, you don't want it to happen with MQ system.

My suggestion would be to stretch the abstraction of ignoring the 
internal node state/algorithms to fully cover the node behaviour:

1. Node is a named endpoint for links and that's it.

2. There's no concept of message being resident on a node.

3. There's no message ordering defined. This follows from the above.

4. There's no message state machine. This follows from 2.

3. There are no destructive/non-destructive links, just "links".

Adopting the points above would make core specification much cleaner. 
The burden of defining storage/ordering/routing etc. is left to "broker 
behaviour" specification.

Martin