I've ran into something odd. Previously, using RabbitMQ 2.8.5 and R13B03 (the default Erlang for Ubunto 10.04) I had three brokers (mq1, mq2, mq3) in a symmetric exchange configuration (e.g, mq1 was the upstream for mq2/mq3, and vice-versa.)<br>
<br>Everything worked groovy and I could connect to all the brokers, I could pump messages through my defined topic exchange ('skytap') between federated brokers, and so forth.<br><br>After upgrading my Erlang version to R15B01, brokers mq2 and mq3 start fine, work as expected and I see my 'skytap' exchange on them. However, mq1 mostly starts up (I see lots of mnesia spew), but it never accepts connections, I can't log into the management console. The beam.smp process is running on mq3, however.<br>
<br>My config file for mq1 looks like this:<br>[<br> {mnesia, [{debug, trace}<br> ]<br> },<br> {rabbitmq_federation,<br> [ {exchanges, [[{exchange, "skytap"},<br> {type, "topic"},<br>
{upstream_set, "my_upstreams"}]]},<br><br> {upstream_sets, [{"my_upstreams", [[{connection, "upstream-server-mq2"},<br> {exchange, "skytap"}],<br>
[{connection, "upstream-server-mq3"},<br> {exchange, "skytap"}]<br> ]}<br>
]},<br> {connections, [{"upstream-server-mq2", [{host, "mq2"}<br> ]<br> },<br> {"upstream-server-mq3", [{host, "mq3"}]<br>
}<br> ]<br> }<br> ]<br> }<br>].<br><br>Whereas for both mq2 and mq3:<br><br>[<br> {mnesia, [{debug, trace}<br> ]<br> },<br> {rabbitmq_federation,<br>
[ {exchanges, [[{exchange, "skytap"},<br> {type, "topic"},<br> {upstream_set, "my_upstreams"}]]},<br><br> {upstream_sets, [{"my_upstreams", [[{connection, "upstream-server"},<br>
{exchange, "skytap"}]<br> ]}<br> ]},<br> {connections, [{"upstream-server", [{host, "mq1"}<br>
]}<br> ]}<br> ]<br> }<br>].<br><br>The rabbit@mq1.log file has this:<br>=INFO REPORT==== 21-Aug-2012::17:09:23 ===<br>Limiting to approx 924 file handles (829 sockets)<br>
<br>=INFO REPORT==== 21-Aug-2012::17:09:23 ===<br>Memory limit set to 399MB of 999MB total.<br><br>=INFO REPORT==== 21-Aug-2012::17:09:23 ===<br>Disk free limit set to 953MB<br><br>=WARNING REPORT==== 21-Aug-2012::17:09:23 ===<br>
Could not find exchange type x-federation.<br><br>=WARNING REPORT==== 21-Aug-2012::17:09:23 ===<br>Could not find exchange type x-federation.<br><br>=INFO REPORT==== 21-Aug-2012::17:09:23 ===<br>msg_store_transient: using rabbit_msg_store_ets_index to provide index<br>
<br>=INFO REPORT==== 21-Aug-2012::17:09:23 ===<br>msg_store_persistent: using rabbit_msg_store_ets_index to provide index<br><br>=WARNING REPORT==== 21-Aug-2012::17:09:23 ===<br>Could not find exchange type x-federation.<br>
<br>=INFO REPORT==== 21-Aug-2012::17:09:23 ===<br>started TCP Listener on <a href="http://0.0.0.0:5672">0.0.0.0:5672</a><br><br>=INFO REPORT==== 21-Aug-2012::17:09:24 ===<br>accepting AMQP connection <0.212.0> (<a href="http://192.168.78.18:55219">192.168.78.18:55219</a> -> <a href="http://192.168.78.16:5672">192.168.78.16:5672</a>)<br>
<br>=INFO REPORT==== 21-Aug-2012::17:09:24 ===<br>accepting AMQP connection <0.219.0> (<a href="http://192.168.78.17:58709">192.168.78.17:58709</a> -> <a href="http://192.168.78.16:5672">192.168.78.16:5672</a>)<br>
<br>=ERROR REPORT==== 21-Aug-2012::17:09:24 ===<br>connection <0.212.0>, channel 2 - error:<br>{amqp_error,not_found,<br> "no exchange 'federation: skytap -> <a href="mailto:rabbit@mq3.mpietrek.internal.illumita.com">rabbit@mq3.mpietrek.internal.illumita.com</a> B' in vhost '/'",<br>
'exchange.delete'}<br><br>=INFO REPORT==== 21-Aug-2012::17:09:24 ===<br>accepting AMQP connection <0.233.0> (<a href="http://192.168.78.18:50211">192.168.78.18:50211</a> -> <a href="http://192.168.78.16:5672">192.168.78.16:5672</a>)<br>
<br>=ERROR REPORT==== 21-Aug-2012::17:09:24 ===<br>connection <0.219.0>, channel 2 - error:<br>{amqp_error,not_found,<br> "no exchange 'federation: skytap -> <a href="mailto:rabbit@mq2.mpietrek.internal.illumita.com">rabbit@mq2.mpietrek.internal.illumita.com</a> A' in vhost '/'",<br>
'exchange.delete'}<br><br>=ERROR REPORT==== 21-Aug-2012::17:09:24 ===<br>connection <0.233.0>, channel 1 - error:<br>{amqp_error,command_invalid,"unknown exchange type 'x-federation-upstream'",<br>
'exchange.declare'}<br><br>=INFO REPORT==== 21-Aug-2012::17:09:24 ===<br>accepting AMQP connection <0.245.0> (<a href="http://192.168.78.17:58290">192.168.78.17:58290</a> -> <a href="http://192.168.78.16:5672">192.168.78.16:5672</a>)<br>
<br>=INFO REPORT==== 21-Aug-2012::17:09:24 ===<br>closing AMQP connection <0.233.0> (<a href="http://192.168.78.18:50211">192.168.78.18:50211</a> -> <a href="http://192.168.78.16:5672">192.168.78.16:5672</a>)<br>
<br>=WARNING REPORT==== 21-Aug-2012::17:09:24 ===<br>Could not find exchange type x-federation.<br><br>=ERROR REPORT==== 21-Aug-2012::17:09:24 ===<br>connection <0.245.0>, channel 1 - error:<br>{amqp_error,command_invalid,"unknown exchange type 'x-federation-upstream'",<br>
'exchange.declare'}<br><br>=INFO REPORT==== 21-Aug-2012::17:09:24 ===<br>closing AMQP connection <0.245.0> (<a href="http://192.168.78.17:58290">192.168.78.17:58290</a> -> <a href="http://192.168.78.16:5672">192.168.78.16:5672</a>)<br>
<br>=WARNING REPORT==== 21-Aug-2012::17:09:24 ===<br>Could not find exchange type x-federation.<br><br>=WARNING REPORT==== 21-Aug-2012::17:09:24 ===<br>Could not find exchange type x-federation.<br><br>=WARNING REPORT==== 21-Aug-2012::17:09:24 ===<br>
Could not find exchange type x-federation.<br><br>----<br>And rabbit@mq2.log:<br><br>=WARNING REPORT==== 21-Aug-2012::17:09:20 ===<br>Connection (<0.300.0>) closing: received hard error {'connection.close',320,<br>
<<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>,<br> 0,0} from server<br>
<br>=INFO REPORT==== 21-Aug-2012::17:09:20 ===<br>Federation exchange 'skytap' in vhost '/' disconnected from mq1:/:skytap<br>{upstream_channel_down,<br> {connection_closing,<br> {server_initiated_close,320,<br>
<<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>}}}<br><br>=ERROR REPORT==== 21-Aug-2012::17:09:20 ===<br>** Generic server <0.300.0> terminating <br>
** Last message in was {channel_exit,3,{writer,send_failed,{error,closed}}}<br>** When Server state == {state,amqp_network_connection,<br> {state,#Port<0.5842>,0,<0.304.0>,131072,<br>
{server_initiated_close,320,<br> <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>},<br> true},<br>
<0.299.0>,<0.302.0>,<br> {amqp_params_network,<<"guest">>,<<"guest">>,<br> <<"/">>,"mq1",5672,0,0,0,infinity,none,<br>
[#Fun<amqp_auth_mechanisms.plain.3>,<br> #Fun<amqp_auth_mechanisms.amqplain.3>],<br> [],[]},<br> 0,<br>
[{<<"capabilities">>,table,<br> [{<<"publisher_confirms">>,bool,true},<br> {<<"exchange_exchange_bindings">>,bool,true},<br>
{<<"basic.nack">>,bool,true},<br> {<<"consumer_cancel_notify">>,bool,true}]},<br> {<<"copyright">>,longstr,<br>
<<"Copyright (C) 2007-2012 VMware, Inc.">>},<br> {<<"information">>,longstr,<br> <<"Licensed under the MPL. See <a href="http://www.rabbitmq.com/">http://www.rabbitmq.com/</a>">>},<br>
{<<"platform">>,longstr,<<"Erlang/OTP">>},<br> {<<"product">>,longstr,<<"RabbitMQ">>},<br>
{<<"version">>,longstr,<<"2.8.5">>}],<br> #Fun<amqp_connection_sup.0.39273983>,<br> #Fun<amqp_connection_sup.2.54430129>,<br>
{closing,server_initiated_close,<br> {'connection.close',320,<br> <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>,<br>
0,0},<br> none}}<br>** Reason for termination == <br>** {function_clause,<br> [{amqp_network_connection,handle_message,<br> [{channel_exit,3,{writer,send_failed,{error,closed}}},<br>
{state,#Port<0.5842>,0,<0.304.0>,131072,<br> {server_initiated_close,320,<br> <<"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'">>},<br>
true}],<br> []},<br> {amqp_gen_connection,callback,3,[]},<br> {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},<br> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}<br>
<br>=ERROR REPORT==== 21-Aug-2012::17:09:20 ===<br>** Generic server <0.305.0> terminating <br>** Last message in was {inet_async,#Port<0.5842>,404,<br> {ok,<<0,10,0,50,1,64,75,67,79,78,78,69,67,84,73,<br>
79,78,95,70,79,82,67,69,68,32,45,32,98,114,<br> 111,107,101,114,32,102,111,114,99,101,100,<br> 32,99,111,110,110,101,99,116,105,111,110,32,<br>
99,108,111,115,117,114,101,32,119,105,116,<br> 104,32,114,101,97,115,111,110,32,39,115,104,<br> 117,116,100,111,119,110,39,0,0,0,0,206>>}}<br>
** When Server state == {state,#Port<0.5842>,<0.300.0>,<0.302.0>,<br> {method,rabbit_framing_amqp_0_9_1},<br> none}<br>** Reason for termination == <br>
** {socket_error,einval}<br><br>=WARNING REPORT==== 21-Aug-2012::17:09:20 ===<br>Federation exchange 'skytap' in vhost '/' did not connect to mq1:/:skytap<br>{error,econnrefused}<br><br>=WARNING REPORT==== 21-Aug-2012::17:09:22 ===<br>
Federation exchange 'skytap' in vhost '/' did not connect to mq1:/:skytap<br>{error,econnrefused}<br><br>=WARNING REPORT==== 21-Aug-2012::17:09:24 ===<br>Connection (<0.412.0>) closing: received hard error {'connection.close',503,<br>
<<"COMMAND_INVALID - unknown exchange type 'x-federation-upstream'">>,<br> 40,10} from server<br>
<br>=INFO REPORT==== 21-Aug-2012::17:09:24 ===<br>Federation exchange 'skytap' in vhost '/' connected to mq1:/:skytap<br><br>---<br>The management console for mq2 shows this on the Exchanges tab:<br><table class="list">
<tbody><tr class="alt1"><td><a href="http://mq2.mpietrek.internal.illumita.com:55672/#/exchanges/%2F/federation%3A%20skytap%20-%3E%20rabbit%40mq1.mpietrek.internal.illumita.com%20A">federation: skytap -> rabbit@mq1.mpietrek.internal.illumita.com A</a></td>
<td class="status c">x-federation-upstream</td>
<td class="c"><acronym title="durable: true">D</acronym> <acronym title="auto-delete: true">AD</acronym> <acronym title="internal: true">I</acronym> <acronym title="x-max-hops: 1">Args</acronym></td>
<td class="r"><br></td></tr></tbody></table><br>And mq3's Exchanges tab shows:<br><table class="list"><tbody><tr class="alt1"><td><a href="http://mq3.mpietrek.internal.illumita.com:55672/#/exchanges/%2F/federation%3A%20skytap%20-%3E%20rabbit%40mq1.mpietrek.internal.illumita.com%20B">federation: skytap -> rabbit@mq1.mpietrek.internal.illumita.com B</a></td>
<td class="status c">x-federation-upstream</td>
<td class="c"><acronym title="durable: true">D</acronym> <acronym title="auto-delete: true">AD</acronym> <acronym title="internal: true">I</acronym> <acronym title="x-max-hops: 1">Args</acronym></td>
<td class="r"><br></td></tr></tbody></table><br><br>Ideas on what I'm doing wrong, or if this is a known issue?<br><br>Thanks,<br><br>Matt<br>