Jiri<div><br></div><div>It makes some sense, but you *will* need to migrate resources, or recreate them, to continue working on the secondary. �You will not need to migrate messages, but there is other broker state that you will need: for example a consumer's exclusive queue on the primary, will need to exist on the secondary, with the correct bindings, connection, etc.</div>
<div><br></div><div>So my question is: do you plan to create that queue (etc) on the secondary after the primary fails? �If "yes" then a potential issue is managing the failover window size and (possible, but less likely) resource limits on the secondary. �If "no" then you will need to 'pre build' the spare resources somehow. �</div>
<div><br></div><div>These are not necessarily difficult issues to solve in any particular instance, however I am highlighting them in response to your request for feedback. �These issues do become deeper when you try to generalise to multiple different failure scenarios, of course.</div>
<div><br></div><div>alexis</div><div><br></div><div><br><div class="gmail_quote">On Fri, Sep 3, 2010 at 8:40 AM, <span dir="ltr"><<a href="mailto:jiri@krutil.com">jiri@krutil.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Alexis<br>
<br>
Our plan is to have two brokers with the same setup, both being actively used at the same time, but each broker serving a different set of clients.<br>
<br>
Our back-end service will be connected to and process requests from both brokers at the same time.<br>
<br>
When one broker fails, the clients will loose connection and will have to reconnect, ending up on the other broker. Messages that were on the dead broker will be lost.<br>
<br>
So we don't need migration of resources between brokers in case of failure, we only need the clients to move their connections to the other broker.<br>
<br>
Hope that makes sense...<br>
<br>
Jiri<div><div></div><div class="h5"><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Jiri<br>
<br>
Cool. �So yes messages will then only arrive out of order in the case<br>
where some arrive from the secondary before 'delayed' messages from<br>
the failed primary; and then, for reordering them, it suffices to know<br>
which broker they came from. �(In the absence of failure, TCP should<br>
take care of reordering).<br>
<br>
I think the issues will be:<br>
<br>
1. Deciding when to stop listening to a primary. �Given consumers<br>
don't care about message loss, I would suggest "as soon as consumers<br>
are aware of primary failure, then they should ignore further messages<br>
from the primary"<br>
<br>
2. Failover time. �AIUI you want to minimise this by having a copy of<br>
the whole queue/exchange/binding set-up on both brokers. �But how<br>
exactly do you plan to do this?<br>
<br>
alexis<br>
<br>
<br>
On Fri, Sep 3, 2010 at 8:12 AM, �<<a href="mailto:jiri@krutil.com" target="_blank">jiri@krutil.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Alexis<br>
<br>
The answer is no - a client can send requests to only one broker at any<br>
given moment. The client connects via load balancer to one of the brokers<br>
and stays connected all the time. The client does not even know that there<br>
are two brokers (it only sees one IP address).<br>
<br>
I think requests may be delivered out of order only if a client fails over<br>
to another broker. Then messages send to one broker can get mixed up with<br>
messages sent to the other.<br>
<br>
My concern was: are there any other issues with this kind of setup that I<br>
might have missed? Does anyone have experience with this?<br>
<br>
Thanks a lot for your help<br>
Jiri<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Jiri<br>
<br>
You say that "Some clients send requests to one broker, some to the<br>
other".<br>
<br>
<br>
Does this mean that one client publisher can send messages (requests) to<br>
both brokers, in such a way that a pair of messages may arrive out of<br>
order<br>
if one is sent to each broker?<br>
<br>
If the answer is no, then I think my answer stands, because causal order<br>
will be preserved even if messages are lost. �That is: messages that<br>
arrive<br>
successfully, will not be out of order with each other.<br>
<br>
If the answer is yes, then I am not sure how you can recover global<br>
ordering<br>
without imposing it at the publisher using sequence numbers at the app<br>
level.<br>
<br>
Does this make sense?<br>
<br>
alexis<br>
<br>
<br>
<br>
On Thu, Sep 2, 2010 at 9:46 PM, Jiri Krutil <<a href="mailto:jiri@krutil.com" target="_blank">jiri@krutil.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Alexis<br>
<br>
Sorry I probably didn't express myself well.<br>
<br>
We don't plan a primary and secondary broker, but a pair of brokers that<br>
are both active at the same time. A load balancer divides client<br>
connections<br>
to these brokers. A request queue with the same name exists on both<br>
brokers,<br>
but with different contents. Some clients send requests to one broker,<br>
some<br>
to the other. Our back-end server listens to both queues, processes<br>
requests<br>
and sends each response to an exclusive client queue on the broker from<br>
where the request came.<br>
<br>
Ideally this would be transparent to the clients, because the brokers<br>
would<br>
be hidden by a virtual IP address. Of course it can't be transparent to<br>
the<br>
back-end server, which needs to talk to both brokers at the same time.<br>
<br>
So (a) is correct, but (b) not.<br>
<br>
Hope that makes it a bit clearer...<br>
<br>
Cheers<br>
Jiri<br>
<br>
<br>
<br>
�Jiri<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
That answered my questions. �Now, as I understood your example:<br>
<br>
a. you don't mind messages being lost<br>
*and*<br>
b. you don't use the secondary until after the primary has failed.<br>
<br>
Note that if consumption is completely 'fire and forget' then it is<br>
possible that a message from the primary may *arrive* after a message<br>
from<br>
the secondary. �But this can happen whether you use sequence numbers or<br>
not.<br>
<br>
So if the primary broker fails, why not just forget all undelivered<br>
messages? �Consumers will know that any message consumed from the<br>
secondary<br>
must be later in *all* orderings than any message consumed from the<br>
primary.<br>
�So, additional sequence numbering is not necessary.<br>
<br>
alexis<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
<br>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
<br>
</div></div></blockquote></div><br></div>