[rabbitmq-discuss] request for help!

[Tim Fox]

On 10/05/10 18:25, Rafael Schloming wrote:
> Tim Fox wrote:
>> On 10/05/10 16:18, Robert Godfrey wrote:
>>> Just to be clear, while this behaviour is permitted under the spec, it
>>> is not mandated that every message exchange follows this pattern.
>>> Firstly the protocol will support different reliability guarantees
>>> agreed at the link level (at most once, at least once, exactly once,
>>> etc) which will allow simpler patterns where the extra guarantees are
>>> not required.
>> I believe you can use a simpler pattern, but still retain the once 
>> and only once guarantee.
>>
>> This is how we do it in HornetQ to give us once and only once when 
>> bridging messages from one server A to another B.
>>
>> The sender doesn't maintain any delivery map at all. Each message has 
>> a de-duplication-id which can either be set by the client when the 
>> message was originally sent, or can be set by the server and derived 
>> from the message and node id of the sender.
>>
>> Since the id can be derived from the message, or is already persisted 
>> as part of the message there's no need to persist it separately or 
>> maintain a map on the client side.
>
> The specification doesn't require you to persist the delivery-tag 
> separately. It's a perfectly valid (and expected) choice to populate 
> it from the message itself as you describe. When you do this the set 
> of unacked messages has all the necessary information to construct the 
> protocol primitives defined in the spec without maintaining a separate 
> map.
>
>> The receiving node B simply maintains a circular cache of received 
>> ids (which is optionally persisted). If B sees the same 
>> de-duplication-id more than once it simply ignores the message.
>
> As I believe Rob already pointed out, this is all the information 
> necessary to fulfill the protocol contracts from the Receiver's side.
>
>> After failure, the sender can just carry on sending any unacked 
>> messages as normal and the server will reject any dups.
>>
>> B sends acks back to A asynchronously in a different stream (so 
>> everything is pipelined for performance). When A receives the ack, 
>> then the message can be removed from storage.
>>
>> This gives us once and only once without having the c) interaction 
>> above.
>
> How does B know when A has received the ack?
>
> If this fact is communicated on the wire, then it is equivalent to the 
> c) interaction. If this fact isn't communicated on the wire, then B is 
> making an assumption when it purges entries from its cache, and is 
> exposing itself to duplicate messages, i.e. this is only providing a 
> probabilistic exactly-once guarantee.
You can tune the duplicate id cache size so that it more than covers the 
amount of unacked messages you might reasonably expect at any one time. 
This figure should be close to the flow control window size. So take the 
figure and multiply by two (or 5 or 10). This should reduce the 
probability of dups to effectively zero.
>
>> I believe it this gives us one less transfer on the wire, but more 
>> importantly one less write to storage - on the sender side.
>
> The c) interaction has almost no overhead since it is trivially 
> inferred from the window of unsettled deliveries which is piggybacked 
> on message transfers, however both the Sender and Receiver can short 
> circuit the full transfer interaction by indicating that a delivery is 
> settled.
>
> This can be done on the initial transfer if fire and 
> forget/at-most-once messaging is desired, or on the initial ack if 
> at-least-once (or probabilistic exactly-once) messaging is desired.
>
>> I can see that your scheme accomplishes the same goal, but it seems 
>> somewhat more complex, unless I am missing something (quite possible 
>> ;) )
>
> I think perhaps the key point is that the spec is defining the 
> protocol primitives and minimal semantics required for interop, but 
> not actually attempting to prescribe or proscribe a particular 
> implementation strategy. So the intention is very much to permit the 
> sort of things you're describing, but not to constrain implementations 
> to choose only those strategies.
>
> As I mentioned to Rob in another post, we could probably be a bit 
> clearer on this in our descriptions and examples.
>
> --Rafael


-- 
Sent from my BBC Micro Model B

Tim Fox
JBoss

HornetQ - putting the buzz in messaging http://hornetq.org
http://hornetq.blogspot.com/
http://twitter.com/hornetq
irc://irc.freenode.net:6667#hornetq	
fox at redhat.com