Hi,<div><br><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I'm running RabbitMQ 3.0.1 on two cluster of Linux servers.<br>
<br>
Let's name the two clusters :<br>
- downstream cluster running a federation to get messages from upstream<br>
cluster<br>
- upstream cluster<br>
<br>
The documentation explains well if a node fails, links to upstream<br>
exchanges will be recreated on a surviving node.<br>
There is no problem for the "client" side of the federation.<br>
<br>
I cannot use a load balancer if fail over mode to have high avaibility<br>
of the upstream cluster.<br>
<br>
What would be the recommended solution in this case ?<br>
<br>
</blockquote>
<br></div>
I'm struggling to understand what the question is here Vladislav. The 'failover' that is being described in the federation plugin documentation is applied when using federation in a cluster, so if the node on which the downstream link is running dies, then another downstream node will take over (i.e., re-establish the links). There is a choice between clustering (i.e., ha/mirror queues) and federation - you do not get 'ha of the upstream cluster' in the same sense that mirror queues in a cluster are 'ha'. You have federated exchanges which copy data using AMQP (with ACKs enabled and some other guarantees) and the ability to try and re-establish links and so. Federation however, provides only the Availability and Partition tolerance parts of the CAP theorem, not the same Consistency guarantees as clustering/ha.</blockquote>
<div><br></div><div>I did get that, no problem for the downstream side who hold the federation, it works well.</div><div>Question is really : I have two nodes in the "remote" or upstream cluster, I want to get messages of one exchange in a reliable way and the network stream has to be establish by the downstream cluster ; how can I be tolerant to failure of one remote node ? That is what I have called high avaibility on the upstream cluster, but only regarding the transmission of the exchange messages to the downstream cluster.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have tried to set up two upstream and group them in a upstream set,<br>
</blockquote>
<br></div>
Can you post the configuration you're using to do that?</blockquote><div><br></div><div>The configuration has been done using the HTTP API.</div><div><br></div><div><div>curl -i -w %{http_code} -k -u "XXX:XXXX" -XPUT -H "content-type:application/json" -d '{</div>
<div>"pattern":"downstream-exchange","definition":{"federation-upstream-set":"upstreamset-test"}</div><div>}' <a href="https://localhost:15671/api/policies/%2f/federate-me">https://localhost:15671/api/policies/%2f/federate-me</a></div>
</div><div><br></div><div><div>curl -i -w %{http_code} -k -u "XXX:XXXX" -XPUT -H "content-type:application/json" -d '{</div><div>"name":"local-nodename","value":"federation-local"</div>
<div>}' <a href="https://localhost:15671/api/parameters/federation/%2f/local-nodename">https://localhost:15671/api/parameters/federation/%2f/local-nodename</a></div></div><div><br></div><div><div>curl -i -w %{http_code} -u "XXX:XXXX" -k -s -XPUT -H "content-type:application/json" -d "{</div>
<div> "value":{</div><div> "uri":\"amqps://XXX:XXX@remote-server1?certfile=XXXXX&keyfile=XXXX&verify=verify_none&fail_if_no_peer_cert=false"</div><div> }</div>
<div> }" <a href="https://localhost:15671/api/parameters/federation-upstream/%2f/upstream1">https://localhost:15671/api/parameters/federation-upstream/%2f/upstream1</a></div></div><div><br></div><div><div>curl -i -w %{http_code} -u "XXX:XXXX" -k -s -XPUT -H "content-type:application/json" -d "{</div>
<div> "value":{</div><div> "uri":\"amqps://XXX:XXX@remote-server2?certfile=XXXXX&keyfile=XXXX&verify=verify_none&fail_if_no_peer_cert=false"</div><div> }</div>
<div> }" <a href="https://localhost:15671/api/parameters/federation-upstream/%2f/upstream2">https://localhost:15671/api/parameters/federation-upstream/%2f/upstream2</a></div></div><div><br></div><div><div>curl -i -w %{http_code} -k -u "XXX:XXXX" -XPUT -H "content-type:application/json" -d '{</div>
<div>"value":[{"upstream":"upstream1","exchange":"upstream-exchange"},{"upstream":"upstream2","exchange":"upstream-exchange"}]</div>
<div>}' <a href="https://localhost:15671/api/parameters/federation-upstream-set/%2f/upstreamset-test">https://localhost:15671/api/parameters/federation-upstream-set/%2f/upstreamset-test</a></div></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
but I have the following problem :<br>
- when I shut down one the node, the federation status shows the<br>
matching upstream down as expected but after having restarted the first<br>
one, if I shut down the other one, both the federation status shows both<br>
upstream down<br>
</blockquote>
<br></div>
Just to confirm: you're saying that<br>
<br>
1. you shut down one of the two upstream nodes<br>
2. that node shows up dead in the web interface<br>
3. you re-start that node<br>
4. that node shows up alive in the web interface<br>
5. you shut down the other upstream node<br>
6. both nodes show up as dead in the web interface *but*<br>
7. one of the upstream nodes *is* alive despite what the web admin says<br>
<br>
Have I understood that correctly?</blockquote><div><br></div><div>Absolutely, you can find an extract of the downstream node log (rabbit@XXXX.log) at the end of the mail.</div><div>That is not that easy to read but I think there might be an explanation in the logs.</div>
<div><br></div><div>Case 2 :</div><div><div>web admin says :</div><div>- upstream1 : running</div><div>- upstream 2 : error (econnrefused)</div></div><div>Everything is OK</div><div><br></div><div>Case 6 :</div><div>web admin says :</div>
<div>- upstream1 : error (econnrefused)</div><div>- upstream 2 : shutdown (server_initiated_code,404,<<"NOT_FOUND ...</div><div>the upstream1 (remote-server1) has been shutdown, but not the upstream2 (remote-server2).</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- so, I tried to add a ha-mode policy to all on the federated queue, it<br>
is now possible to shutdown either one or the other node,<br>
</blockquote>
<br></div>
I'm not sure I understand this at all. Are you saying it was not possible to shut down one or both of the upstream nodes before? That seems different from your earlier comment.</blockquote><div><br></div><div>My bad ! It is not explained properly. When I was saying "not possible to shutdown", I meant shutting down the remote node and having the proper status in the federation.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
but it seems that I'm losing some messages.<br>
<br>
</blockquote>
<br></div>
When you say 'the federated queue' do you mean the queue created in the upstream exchange's broker? Why would you want to add ha-mode policy that? The upstream queue is internal to the federation mechanism so you should be binding to the downstream exchange only. Or are you saying that you've bound a queue to the downstream exchange and made that ha-enabled? Because in the latter case, that will make no difference to reliability: if both upstream nodes go down before messages are delivered and ack'ed by the downstream for example.<br>
</blockquote><div><br></div><div>Yes, "federated queue" is the queue created in the upstream exchange's broker. So, yes, it does not really make sense to add a ha policy, that was just an attempt in order to investigate a little bit further.</div>
<div>And yes I did bound a ha queue to the downstream exchange, but I agree, it has nothing to do with the subject.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I'd be interested to hear how you've set this ha-mode policy and why and also how you've determined that there was message loss? I suspect that you have assumed expectations about the reliability of federation (in the face of node failures) that do not hold. If your messages sat in an exchange on an upstream node (or pair of exchanges/nodes, etc) and both nodes die before successfully transmitting the messages, then they will not arrive at the downstream exchange. The guarantees about message delivery for ha/mirror queues apply to nodes in *that* cluster only. The federation guarantees are different and orthogonal to ha/clustering.<br>
</blockquote><div><br></div><div>I understand, I just want to make sure that the messages will arrive at the downstream exchange, not duplicated and without loss :</div><div>- if one upstream node dies</div><div>- if there is network failure between downstream and upstream nodes</div>
<div>- if upstream nodes fail and come back again</div><div><br></div><div>Thanks.</div><div>Vlad</div><div><br></div><div><b>Extract of logs for case 2</b> :</div><div><br></div><div><div>=WARNING REPORT==== 28-Dec-2012::10:27:08 ===</div>
<div>Connection (<0.14803.351>) closing: received hard error {'connection.close',</div><div> 320,</div><div> <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>,</div>
<div> 0,0} from server</div><div><br></div><div>=ERROR REPORT==== 28-Dec-2012::10:27:08 ===</div><div>** Generic server <0.14803.351> terminating</div><div>** Last message in was {#Ref<0.0.127.4786>,{error,closed}}</div>
<div>** When Server state == {state,amqp_network_connection,</div><div> {state,</div><div> {ssl_socket,#Port<0.46383>,</div><div> {sslsocket,new_ssl,<0.14806.351>}},</div>
<div> 600,<0.14808.351>,131072,</div><div> {server_initiated_close,320,</div><div> <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>},</div>
<div> false},</div><div> <0.14802.351>,<0.14805.351>,</div><div> {amqp_params_network,<<"XXXX">>,</div>
<div> <<"XXXX">>,<<"/">>,"remote-server2",5671,0,</div><div> 0,0,infinity,</div><div> [{fail_if_no_peer_cert,false},</div>
<div> {verify,verify_none},</div><div> {keyfile,</div><div> "XXXX"},</div><div> {certfile,</div>
<div> "XXXX"}],</div><div> [#Fun<amqp_uri.7.123484526>,</div><div> #Fun<amqp_uri.7.123484526>],</div>
<div> [],[]},</div><div> 0,</div><div> [{<<"capabilities">>,table,</div><div> [{<<"publisher_confirms">>,bool,true},</div>
<div> {<<"exchange_exchange_bindings">>,bool,true},</div><div> {<<"basic.nack">>,bool,true},</div><div> {<<"consumer_cancel_notify">>,bool,true}]},</div>
<div> {<<"copyright">>,longstr,</div><div> <<"Copyright (C) 2007-2012 VMware, Inc.">>},</div><div> {<<"information">>,longstr,</div>
<div> <<"Licensed under the MPL. See <a href="http://www.rabbitmq.com/">http://www.rabbitmq.com/</a>">>},</div><div> {<<"platform">>,longstr,<<"Erlang/OTP">>},</div>
<div> {<<"product">>,longstr,<<"RabbitMQ">>},</div><div> {<<"version">>,longstr,<<"3.0.1">>}],</div>
<div> #Fun<amqp_connection_sup.0.39273983>,</div><div> #Fun<amqp_connection_sup.2.54430129>,</div><div> {closing,server_initiated_close,</div>
<div> {'connection.close',320,</div><div> <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>,</div>
<div> 0,0},</div><div> none}}</div><div>** Reason for termination ==</div><div>** socket_closed_unexpectedly</div><div><br></div><div>=INFO REPORT==== 28-Dec-2012::10:27:08 ===</div>
<div>Federation exchange 'downstream-exchange' in vhost '/' disconnected from exchange 'upstream-exchange' in vhost '/' on amqps://XXXX:XXXX@remote-server2?certfile=XXXX&keyfile=XXXX&verify=verify_none&fail_if_no_peer_cert=false</div>
<div>{upstream_channel_down,</div><div> {connection_closing,</div><div> {server_initiated_close,320,</div><div> <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>}}}</div>
<div><br></div><div>=WARNING REPORT==== 28-Dec-2012::10:27:08 ===</div><div>Federation exchange 'downstream-exchange' in vhost '/' did not connect to exchange 'upstream-exchange' in vhost '/' on amqps://XXXX:XXXX@remote-server2?certfile=XXXX&keyfile=XXXX&verify=verify_none&fail_if_no_peer_cert=false</div>
<div>{error,econnrefused}</div><div><br></div><div>==> rabbit@XXXX-sasl.log <==</div><div><br></div><div>=CRASH REPORT==== 28-Dec-2012::10:27:08 ===</div><div> crasher:</div><div> initial call: amqp_gen_connection:init/1</div>
<div> pid: <0.14803.351></div><div> registered_name: []</div><div> exception exit: socket_closed_unexpectedly</div><div> in function gen_server:terminate/6 (gen_server.erl, line 747)</div><div> ancestors: [<0.14802.351>,amqp_sup,<0.49.0>]</div>
<div> messages: []</div><div> links: [<0.14802.351>]</div><div> dictionary: []</div><div> trap_exit: true</div><div> status: running</div><div> heap_size: 2584</div><div> stack_size: 24</div><div>
reductions: 1786</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 28-Dec-2012::10:27:08 ===</div><div> Supervisor: {<0.14802.351>,amqp_connection_sup}</div><div> Context: child_terminated</div>
<div> Reason: socket_closed_unexpectedly</div><div> Offender: [{pid,<0.14803.351>},</div><div> {name,connection},</div><div> {mfa,</div><div> {amqp_gen_connection,start_link,</div>
<div> [amqp_network_connection,</div><div> {amqp_params_network,<<"XXXX">>,</div><div> <<"XXXX">>,<<"/">>,"remote-server2",5671,0,0,</div>
<div> 0,infinity,</div><div> [{fail_if_no_peer_cert,false},</div><div> {verify,verify_none},</div><div> {keyfile,</div>
<div> "XXXX"},</div><div> {certfile,</div><div> "XXXX"}],</div><div> [#Fun<amqp_uri.7.123484526>,</div>
<div> #Fun<amqp_uri.7.123484526>],</div><div> [],[]},</div><div> #Fun<amqp_connection_sup.0.39273983>,</div><div> #Fun<amqp_connection_sup.2.54430129>,[]]}},</div>
<div> {restart_type,intrinsic},</div><div> {shutdown,brutal_kill},</div><div> {child_type,worker}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 28-Dec-2012::10:27:08 ===</div>
<div> Supervisor: {<0.14802.351>,amqp_connection_sup}</div><div> Context: shutdown</div><div> Reason: reached_max_restart_intensity</div><div> Offender: [{pid,<0.14803.351>},</div><div>
{name,connection},</div><div> {mfa,</div><div> {amqp_gen_connection,start_link,</div><div> [amqp_network_connection,</div><div> {amqp_params_network,<<"XXXX">>,</div>
<div> <<"XXXX">>,<<"/">>,"remote-server2",5671,0,0,</div><div> 0,infinity,</div><div> [{fail_if_no_peer_cert,false},</div>
<div> {verify,verify_none},</div><div> {keyfile,</div><div> "XXXX"},</div><div> {certfile,</div>
<div> "XXXX"}],</div><div> [#Fun<amqp_uri.7.123484526>,</div><div> #Fun<amqp_uri.7.123484526>],</div><div>
[],[]},</div><div> #Fun<amqp_connection_sup.0.39273983>,</div><div> #Fun<amqp_connection_sup.2.54430129>,[]]}},</div><div> {restart_type,intrinsic},</div>
<div> {shutdown,brutal_kill},</div><div> {child_type,worker}]</div><div><br></div><div><br></div><div>==> rabbit@XXXX.log <==</div><div><br></div><div>=WARNING REPORT==== 28-Dec-2012::10:27:10 ===</div>
<div>Federation exchange 'downstream-exchange' in vhost '/' did not connect to exchange 'upstream-exchange' in vhost '/' on amqps://XXXX:XXXX@remote-server2?certfile=XXXX&keyfile=XXXX&verify=verify_none&fail_if_no_peer_cert=false</div>
<div>{error,econnrefused}</div></div><div><br></div><div><br></div><div><b>Extract of logs for case 6</b> :</div><div><br></div><div><div>=WARNING REPORT==== 28-Dec-2012::10:16:42 ===</div><div>Connection (<0.32151.341>) closing: received hard error {'connection.close',</div>
<div> 320,</div><div> <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>,</div>
<div> 0,0} from server</div><div><br></div><div>=ERROR REPORT==== 28-Dec-2012::10:16:42 ===</div><div>** Generic server <0.32151.341> terminating</div><div>** Last message in was {#Ref<0.0.123.217100>,{error,closed}}</div>
<div>** When Server state == {state,amqp_network_connection,</div><div> {state,</div><div> {ssl_socket,#Port<0.45661>,</div><div> {sslsocket,new_ssl,<0.32161.341>}},</div>
<div> 600,<0.32170.341>,131072,</div><div> {server_initiated_close,320,</div><div> <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>},</div>
<div> false},</div><div> <0.32149.341>,<0.32154.341>,</div><div> {amqp_params_network,<<"XXXX">>,</div>
<div> <<"XXXX">>,<<"/">>,"remote-server1",5671,0,</div><div> 0,0,infinity,</div><div> [{fail_if_no_peer_cert,false},</div>
<div> {verify,verify_none},</div><div> {keyfile,</div><div> "XXXX"},</div><div> {certfile,</div>
<div> "XXXX"}],</div><div> [#Fun<amqp_uri.7.123484526>,</div><div> #Fun<amqp_uri.7.123484526>],</div>
<div> [],[]},</div><div> 0,</div><div> [{<<"capabilities">>,table,</div><div> [{<<"publisher_confirms">>,bool,true},</div>
<div> {<<"exchange_exchange_bindings">>,bool,true},</div><div> {<<"basic.nack">>,bool,true},</div><div> {<<"consumer_cancel_notify">>,bool,true}]},</div>
<div> {<<"copyright">>,longstr,</div><div> <<"Copyright (C) 2007-2012 VMware, Inc.">>},</div><div> {<<"information">>,longstr,</div>
<div> <<"Licensed under the MPL. See <a href="http://www.rabbitmq.com/">http://www.rabbitmq.com/</a>">>},</div><div> {<<"platform">>,longstr,<<"Erlang/OTP">>},</div>
<div> {<<"product">>,longstr,<<"RabbitMQ">>},</div><div> {<<"version">>,longstr,<<"3.0.1">>}],</div>
<div> #Fun<amqp_connection_sup.0.39273983>,</div><div> #Fun<amqp_connection_sup.2.54430129>,</div><div> {closing,server_initiated_close,</div>
<div> {'connection.close',320,</div><div> <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>,</div>
<div> 0,0},</div><div> none}}</div><div>** Reason for termination ==</div><div>** socket_closed_unexpectedly</div><div><br></div><div>=ERROR REPORT==== 28-Dec-2012::10:16:42 ===</div>
<div>** Generic server <0.32126.341> terminating</div><div>** Last message in was {'DOWN',#Ref<0.0.123.217187>,process,<0.32193.341>,</div><div> shutdown}</div><div>
** When Server state == {state,</div><div> {upstream,</div><div> {amqp_params_network,<<"XXXX">>,</div><div> <<"XXXX">>,<<"/">>,"remote-server1",undefined,0,</div>
<div> 0,0,infinity,</div><div> [{fail_if_no_peer_cert,false},</div><div> {verify,verify_none},</div><div> {keyfile,</div>
<div> "XXXX"},</div><div> {certfile,</div><div> "XXXX"}],</div><div> [#Fun<amqp_uri.7.123484526>,</div>
<div> #Fun<amqp_uri.7.123484526>],</div><div> [],[]},</div><div> <<"amqps://XXXX:XXXX@remote-server1?certfile=XXXX&keyfile=XXXX&verify=verify_none&fail_if_no_peer_cert=false">>,</div>
<div> {exchange,</div><div> {resource,<<"/">>,exchange,<<"upstream-exchange">>},</div><div> direct,true,false,false,[],undefined,</div>
<div> [{vhost,<<"/">>},</div><div> {name,<<"federate-me">>},</div><div> {pattern,<<"downstream-exchange">>},</div>
<div> {definition,</div><div> [{<<"federation-upstream-set">>,</div><div> <<"upstreamset-test">>}]},</div>
<div> {priority,0}]},</div><div> 1000,1,1,none,none,false,none,<<"upstream1">>},</div><div> <0.32151.341>,<0.32193.341>,</div>
<div> <<"amq.ctag-iCWCgBnLBU7S3cWTi06V1A">>,</div><div> <<"federation: upstream-exchange -> federation-local:downstream-exchange">>,</div>
<div> <<"federation: upstream-exchange -> federation-local:downstream-exchange B">>,</div><div> {0,nil},</div><div> 1,</div>
<div> {dict,1,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> [[{<<"test">>,[]}|</div><div> {set,1,16,16,8,80,48,</div><div> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div>
<div> []},</div><div> {{[],[],[],[],</div><div> [{resource,<<"/">>,queue,</div><div> <<"downstream-queue">>}],</div>
<div> [],[],[],[],[],[],[],[],[],[],[]}}}]]}}},</div><div> <0.32129.341>,<0.32141.341>,</div><div> {resource,<<"/">>,exchange,<<"downstream-exchange">>},</div>
<div> {0,nil}}</div><div>** Reason for termination ==</div><div>** {upstream_channel_down,shutdown}</div><div><br></div><div>=INFO REPORT==== 28-Dec-2012::10:16:42 ===</div><div>Federation exchange 'downstream-exchange' in vhost '/' received 'basic.cancel'</div>
<div><br></div><div>=WARNING REPORT==== 28-Dec-2012::10:16:42 ===</div><div>Federation exchange 'downstream-exchange' in vhost '/' did not connect to exchange 'upstream-exchange' in vhost '/' on amqps://XXXX:XXXX@remote-server1?certfile=XXXX&keyfile=XXXX&verify=verify_none&fail_if_no_peer_cert=false</div>
<div>{error,econnrefused}</div><div><br></div><div>==> rabbit@XXXX-sasl.log <==</div><div><br></div><div>=CRASH REPORT==== 28-Dec-2012::10:16:42 ===</div><div> crasher:</div><div> initial call: amqp_gen_connection:init/1</div>
<div> pid: <0.32151.341></div><div> registered_name: []</div><div> exception exit: socket_closed_unexpectedly</div><div> in function gen_server:terminate/6 (gen_server.erl, line 747)</div><div> ancestors: [<0.32149.341>,amqp_sup,<0.49.0>]</div>
<div> messages: [socket_closed]</div><div> links: [<0.32149.341>]</div><div> dictionary: []</div><div> trap_exit: true</div><div> status: running</div><div> heap_size: 2584</div><div> stack_size: 24</div>
<div> reductions: 1794</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 28-Dec-2012::10:16:42 ===</div><div> Supervisor: {<0.32149.341>,amqp_connection_sup}</div><div> Context: child_terminated</div>
<div> Reason: socket_closed_unexpectedly</div><div> Offender: [{pid,<0.32151.341>},</div><div> {name,connection},</div><div> {mfa,</div><div> {amqp_gen_connection,start_link,</div>
<div> [amqp_network_connection,</div><div> {amqp_params_network,<<"XXXX">>,</div><div> <<"XXXX">>,<<"/">>,"remote-server1",5671,0,0,</div>
<div> 0,infinity,</div><div> [{fail_if_no_peer_cert,false},</div><div> {verify,verify_none},</div><div> {keyfile,</div>
<div> "XXXX"},</div><div> {certfile,</div><div> "XXXX"}],</div><div> [#Fun<amqp_uri.7.123484526>,</div>
<div> #Fun<amqp_uri.7.123484526>],</div><div> [],[]},</div><div> #Fun<amqp_connection_sup.0.39273983>,</div><div> #Fun<amqp_connection_sup.2.54430129>,[]]}},</div>
<div> {restart_type,intrinsic},</div><div> {shutdown,brutal_kill},</div><div> {child_type,worker}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 28-Dec-2012::10:16:42 ===</div>
<div> Supervisor: {<0.32149.341>,amqp_connection_sup}</div><div> Context: shutdown</div><div> Reason: reached_max_restart_intensity</div><div> Offender: [{pid,<0.32151.341>},</div><div>
{name,connection},</div><div> {mfa,</div><div> {amqp_gen_connection,start_link,</div><div> [amqp_network_connection,</div><div> {amqp_params_network,<<"XXXX">>,</div>
<div> <<"XXXX">>,<<"/">>,"remote-server1",5671,0,0,</div><div> 0,infinity,</div><div> [{fail_if_no_peer_cert,false},</div>
<div> {verify,verify_none},</div><div> {keyfile,</div><div> "XXXX"},</div><div> {certfile,</div>
<div> "XXXX"}],</div><div> [#Fun<amqp_uri.7.123484526>,</div><div> #Fun<amqp_uri.7.123484526>],</div><div>
[],[]},</div><div> #Fun<amqp_connection_sup.0.39273983>,</div><div> #Fun<amqp_connection_sup.2.54430129>,[]]}},</div><div> {restart_type,intrinsic},</div>
<div> {shutdown,brutal_kill},</div><div> {child_type,worker}]</div><div><br></div><div><br></div><div>=CRASH REPORT==== 28-Dec-2012::10:16:42 ===</div><div> crasher:</div><div> initial call: gen:init_it/6</div>
<div> pid: <0.32126.341></div><div> registered_name: []</div><div> exception exit: {upstream_channel_down,shutdown}</div><div> in function gen_server2:terminate/3</div><div> ancestors: [<0.32125.341>,<0.218.0>,rabbit_federation_link_sup_sup,</div>
<div> rabbit_federation_sup,rabbit_sup,<0.165.0>]</div><div> messages: [{'DOWN',#Ref<0.0.123.217038>,process,<0.32141.341>,normal}]</div><div> links: [<0.32125.341>]</div>
<div> dictionary: []</div><div> trap_exit: true</div><div> status: running</div><div> heap_size: 1597</div><div> stack_size: 24</div><div> reductions: 2291192</div><div> neighbours:</div><div><br></div>
<div>==> rabbit@XXXX.log <==</div><div><br></div><div>=WARNING REPORT==== 28-Dec-2012::10:16:44 ===</div><div>Federation exchange 'downstream-exchange' in vhost '/' did not connect to exchange 'upstream-exchange' in vhost '/' on amqps://XXXX:XXXX@remote-server1?certfile=XXXX&keyfile=XXXX&verify=verify_none&fail_if_no_peer_cert=false</div>
<div>{error,econnrefused}</div><div><br></div><div>=WARNING REPORT==== 28-Dec-2012::10:16:46 ===</div><div>Federation exchange 'downstream-exchange' in vhost '/' did not connect to exchange 'upstream-exchange' in vhost '/' on amqps://XXXX:XXXX@remote-server2?certfile=XXXX&keyfile=XXXX&verify=verify_none&fail_if_no_peer_cert=false</div>
<div>{{shutdown,{server_initiated_close,404,</div><div> <<"NOT_FOUND - home node 'rabbit@remote-server1' of durable queue 'federation: upstream-exchange -> federation-local:downstream-exchange' in vhost '/' is down or inaccessible">>}},</div>
<div> {gen_server,call,</div><div> [<0.7847.351>,</div><div> {call,{'queue.declare',0,</div><div> <<"federation: upstream-exchange -> federation-local:downstream-exchange">>,</div>
<div> false,true,false,false,false,[]},</div><div> none,<0.7815.351>},</div><div> infinity]}}</div><div><br></div><div>=WARNING REPORT==== 28-Dec-2012::10:16:48 ===</div>
<div>Federation exchange 'downstream-exchange' in vhost '/' did not connect to exchange 'upstream-exchange' in vhost '/' on amqps://XXXX:XXXX@remote-server1?certfile=XXXX&keyfile=XXXX&verify=verify_none&fail_if_no_peer_cert=false</div>
<div>{error,econnrefused}</div></div><div><br></div></div></div></div>