[rabbitmq-discuss] Fully reliable setup impossible?

Fri May 23 13:03:25 BST 2014

>
>  What I mean when I say "reliable" is: All subscribers at the time of
>> publish will eventually get the message.
>>
>> That should be possible, assuming that all live inconsistent nodes will
>> eventually rejoin (without dumping messages). I know this is not the
>> case in rabbitmq, but it is definitely theoretically possible. I guess
>> this is what is usually referred to as eventual consistency.
>>
>  So why did you originally say:
>
>> It must be a cluster because we want clients to be able to connect to
>> each node transparently. Federation is not an option.
>>
>  ...because it really sounds like federation is what you want :-)
>

Services in our setup must be ignorant to which of the two data centers
they are connected to. Since queues are not mirrored, this is not possible
with a federation.

>  Do you know what the consequence of doubling it might be?
>>
>
> It will take twice as long to conclude that a remote node that is no
> longer connected has actually gone away. Until then, things can block.

Thanks. Perfect.

>  About the CAP theorem in relation to rabbit.
>> Reliable messaging (zero message loss), is often preferred in
>> SOA-settings. I wonder why vmware/pivotal/... chose not to prioritize
>> this guarantee. It is aimed by the federation setup, but it is a little
>> to weak in its synchronization. It would be preferred if it had a
>> possibility of communicating consumption of messages. Then one could
>> mirror queues between up/down-stream exchanges, and have even more
>> "availability".
>>
>
> I'm not sure what you're talking about here.

Sorry for being unclear.

> Federation certainly should be able to ensure zero message loss! (Assuming
> you leave it in "on-confirm" ack-mode).

Yes, assuming that there actually is a queue (with a binding) on the right
node.

> So when you say "if it had a possibility of communicating consumption of
> messages" you're talking about a sort of eventually consistent federated
> mirrored queue? I have wondered about producing such a thing. But as usual
> the list of things we could do is large, and the resources small. And I
> suspect the cost of producing it would be quite large, mostly due to the
> need to somehow reunify the queues after a partition.

This is really exactly what is needed IMO. It sounds very much like this is
the setup TIBCO uses, when guaranteeing zero-message-loss.
I definitely appreciate that the reunification procedure is not simple.
However, mirrored queues are usually somewhat static (in that they are
created once and lives forever). Maybe a simple queue union, with some sort
of message consumption union as well, so node 1 can delete messages
consumed on node 2. I suppose it could be implemented efficiently by adding
a sequence number to the messages in each queue, and then having a list of
intervals of consumed message sequence numbers, or similar.

It would be a great (!) addition to rabbitmq, and I am very interested in
hearing of any progress.

Thank you big time,
-- S :-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140523/22035ea1/attachment.html>