[rabbitmq-discuss] request for help!

Mon May 10 18:25:08 BST 2010

Tim Fox wrote:
> On 10/05/10 16:18, Robert Godfrey wrote:
>> Just to be clear, while this behaviour is permitted under the spec, it
>> is not mandated that every message exchange follows this pattern.
>> Firstly the protocol will support different reliability guarantees
>> agreed at the link level (at most once, at least once, exactly once,
>> etc) which will allow simpler patterns where the extra guarantees are
>> not required.
> I believe you can use a simpler pattern, but still retain the once and 
> only once guarantee.
> 
> This is how we do it in HornetQ to give us once and only once when 
> bridging messages from one server A to another B.
> 
> The sender doesn't maintain any delivery map at all. Each message has a 
> de-duplication-id which can either be set by the client when the message 
> was originally sent, or can be set by the server and derived from the 
> message and node id of the sender.
> 
> Since the id can be derived from the message, or is already persisted as 
> part of the message there's no need to persist it separately or maintain 
> a map on the client side.

The specification doesn't require you to persist the delivery-tag 
separately. It's a perfectly valid (and expected) choice to populate it 
from the message itself as you describe. When you do this the set of 
unacked messages has all the necessary information to construct the 
protocol primitives defined in the spec without maintaining a separate map.

> The receiving node B simply maintains a circular cache of received ids 
> (which is optionally persisted). If B sees the same de-duplication-id 
> more than once it simply ignores the message.

As I believe Rob already pointed out, this is all the information 
necessary to fulfill the protocol contracts from the Receiver's side.

> After failure, the sender can just carry on sending any unacked messages 
> as normal and the server will reject any dups.
> 
> B sends acks back to A asynchronously in a different stream (so 
> everything is pipelined for performance). When A receives the ack, then 
> the message can be removed from storage.
> 
> This gives us once and only once without having the c) interaction above.

How does B know when A has received the ack?

If this fact is communicated on the wire, then it is equivalent to the 
c) interaction. If this fact isn't communicated on the wire, then B is 
making an assumption when it purges entries from its cache, and is 
exposing itself to duplicate messages, i.e. this is only providing a 
probabilistic exactly-once guarantee.

> I believe it this gives us one less transfer on the wire, but more 
> importantly one less write to storage - on the sender side.

The c) interaction has almost no overhead since it is trivially inferred 
from the window of unsettled deliveries which is piggybacked on message 
transfers, however both the Sender and Receiver can short circuit the 
full transfer interaction by indicating that a delivery is settled.

This can be done on the initial transfer if fire and forget/at-most-once 
messaging is desired, or on the initial ack if at-least-once (or 
probabilistic exactly-once) messaging is desired.

> I can see that your scheme accomplishes the same goal, but it seems 
> somewhat more complex, unless I am missing something (quite possible ;) )

I think perhaps the key point is that the spec is defining the protocol 
primitives and minimal semantics required for interop, but not actually 
attempting to prescribe or proscribe a particular implementation 
strategy. So the intention is very much to permit the sort of things 
you're describing, but not to constrain implementations to choose only 
those strategies.

As I mentioned to Rob in another post, we could probably be a bit 
clearer on this in our descriptions and examples.

--Rafael