Jiri<div><br></div><div>It makes some sense, but you *will* need to migrate resources, or recreate them, to continue working on the secondary. You will not need to migrate messages, but there is other broker state that you will need: for example a consumer's exclusive queue on the primary, will need to exist on the secondary, with the correct bindings, connection, etc.</div>
<div><br></div><div>So my question is: do you plan to create that queue (etc) on the secondary after the primary fails? If "yes" then a potential issue is managing the failover window size and (possible, but less likely) resource limits on the secondary. If "no" then you will need to 'pre build' the spare resources somehow. </div>
<div><br></div><div>These are not necessarily difficult issues to solve in any particular instance, however I am highlighting them in response to your request for feedback. These issues do become deeper when you try to generalise to multiple different failure scenarios, of course.</div>
<div><br></div><div>alexis</div><div><br></div><div><br><div class="gmail_quote">On Fri, Sep 3, 2010 at 8:40 AM, <span dir="ltr"><<a href="mailto:jiri@krutil.com">jiri@krutil.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Alexis<br>
<br>
Our plan is to have two brokers with the same setup, both being actively used at the same time, but each broker serving a different set of clients.<br>
<br>
Our back-end service will be connected to and process requests from both brokers at the same time.<br>
<br>
When one broker fails, the clients will loose connection and will have to reconnect, ending up on the other broker. Messages that were on the dead broker will be lost.<br>
<br>
So we don't need migration of resources between brokers in case of failure, we only need the clients to move their connections to the other broker.<br>
<br>
Hope that makes sense...<br>
<br>
Jiri<div><div></div><div class="h5"><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Jiri<br>
<br>
Cool. So yes messages will then only arrive out of order in the case<br>
where some arrive from the secondary before 'delayed' messages from<br>
the failed primary; and then, for reordering them, it suffices to know<br>
which broker they came from. (In the absence of failure, TCP should<br>
take care of reordering).<br>
<br>
I think the issues will be:<br>
<br>
1. Deciding when to stop listening to a primary. Given consumers<br>
don't care about message loss, I would suggest "as soon as consumers<br>
are aware of primary failure, then they should ignore further messages<br>
from the primary"<br>
<br>
2. Failover time. AIUI you want to minimise this by having a copy of<br>
the whole queue/exchange/binding set-up on both brokers. But how<br>
exactly do you plan to do this?<br>
<br>
alexis<br>
<br>
<br>
On Fri, Sep 3, 2010 at 8:12 AM, <<a href="mailto:jiri@krutil.com" target="_blank">jiri@krutil.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Alexis<br>
<br>
The answer is no - a client can send requests to only one broker at any<br>
given moment. The client connects via load balancer to one of the brokers<br>
and stays connected all the time. The client does not even know that there<br>
are two brokers (it only sees one IP address).<br>
<br>
I think requests may be delivered out of order only if a client fails over<br>
to another broker. Then messages send to one broker can get mixed up with<br>
messages sent to the other.<br>
<br>
My concern was: are there any other issues with this kind of setup that I<br>
might have missed? Does anyone have experience with this?<br>
<br>
Thanks a lot for your help<br>
Jiri<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Jiri<br>
<br>
You say that "Some clients send requests to one broker, some to the<br>
other".<br>
<br>
<br>
Does this mean that one client publisher can send messages (requests) to<br>
both brokers, in such a way that a pair of messages may arrive out of<br>
order<br>
if one is sent to each broker?<br>
<br>
If the answer is no, then I think my answer stands, because causal order<br>
will be preserved even if messages are lost. That is: messages that<br>
arrive<br>
successfully, will not be out of order with each other.<br>
<br>
If the answer is yes, then I am not sure how you can recover global<br>
ordering<br>
without imposing it at the publisher using sequence numbers at the app<br>
level.<br>
<br>
Does this make sense?<br>
<br>
alexis<br>
<br>
<br>
<br>
On Thu, Sep 2, 2010 at 9:46 PM, Jiri Krutil <<a href="mailto:jiri@krutil.com" target="_blank">jiri@krutil.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Alexis<br>
<br>
Sorry I probably didn't express myself well.<br>
<br>
We don't plan a primary and secondary broker, but a pair of brokers that<br>
are both active at the same time. A load balancer divides client<br>
connections<br>
to these brokers. A request queue with the same name exists on both<br>
brokers,<br>
but with different contents. Some clients send requests to one broker,<br>
some<br>
to the other. Our back-end server listens to both queues, processes<br>
requests<br>
and sends each response to an exclusive client queue on the broker from<br>
where the request came.<br>
<br>
Ideally this would be transparent to the clients, because the brokers<br>
would<br>
be hidden by a virtual IP address. Of course it can't be transparent to<br>
the<br>
back-end server, which needs to talk to both brokers at the same time.<br>
<br>
So (a) is correct, but (b) not.<br>
<br>
Hope that makes it a bit clearer...<br>
<br>
Cheers<br>
Jiri<br>
<br>
<br>
<br>
Jiri<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
That answered my questions. Now, as I understood your example:<br>
<br>
a. you don't mind messages being lost<br>
*and*<br>
b. you don't use the secondary until after the primary has failed.<br>
<br>
Note that if consumption is completely 'fire and forget' then it is<br>
possible that a message from the primary may *arrive* after a message<br>
from<br>
the secondary. But this can happen whether you use sequence numbers or<br>
not.<br>
<br>
So if the primary broker fails, why not just forget all undelivered<br>
messages? Consumers will know that any message consumed from the<br>
secondary<br>
must be later in *all* orderings than any message consumed from the<br>
primary.<br>
So, additional sequence numbering is not necessary.<br>
<br>
alexis<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
<br>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
<br>
</div></div></blockquote></div><br></div>