[rabbitmq-discuss] Exchange/Federation problems (unknown exchange type 'x-federation-upstream') and inability to connect to a broker after moving from R13B03 to R15B01

Wed Aug 22 01:33:56 BST 2012

I've ran into something odd. Previously, using RabbitMQ 2.8.5 and R13B03
(the default Erlang for Ubunto 10.04) I had three brokers (mq1, mq2, mq3)
in a symmetric exchange configuration (e.g, mq1 was the upstream for
mq2/mq3, and vice-versa.)

Everything worked groovy and I could connect to all the brokers, I could
pump messages through my defined  topic exchange ('skytap') between
federated brokers, and so forth.

After upgrading my Erlang version to R15B01, brokers mq2 and mq3 start
fine, work as expected and I see my 'skytap' exchange on them. However, mq1
mostly starts up (I see lots of mnesia spew), but it never accepts
connections, I can't log into the management console. The beam.smp process
is running on mq3, however.

My config file for mq1 looks like this:
[
    {mnesia, [{debug, trace}
             ]
    },
    {rabbitmq_federation,
        [ {exchanges, [[{exchange, "skytap"},
                        {type, "topic"},
                        {upstream_set, "my_upstreams"}]]},

          {upstream_sets, [{"my_upstreams", [[{connection,
"upstream-server-mq2"},
                                              {exchange, "skytap"}],
                                             [{connection,
"upstream-server-mq3"},
                                              {exchange, "skytap"}]
                                       ]}
                     ]},
          {connections, [{"upstream-server-mq2", [{host, "mq2"}
                                                 ]
                         },
                         {"upstream-server-mq3", [{host, "mq3"}]
                         }
                        ]
          }
        ]
    }
].

Whereas for both mq2 and mq3:

[
    {mnesia, [{debug, trace}
             ]
    },
    {rabbitmq_federation,
        [ {exchanges, [[{exchange, "skytap"},
                        {type, "topic"},
                        {upstream_set, "my_upstreams"}]]},

          {upstream_sets, [{"my_upstreams", [[{connection,
"upstream-server"},
                                         {exchange, "skytap"}]
                                       ]}
                     ]},
          {connections, [{"upstream-server", [{host, "mq1"}
                                        ]}
                   ]}
        ]
    }
].

The rabbit at mq1.log file has this:
=INFO REPORT==== 21-Aug-2012::17:09:23 ===
Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 21-Aug-2012::17:09:23 ===
Memory limit set to 399MB of 999MB total.

=INFO REPORT==== 21-Aug-2012::17:09:23 ===
Disk free limit set to 953MB

=WARNING REPORT==== 21-Aug-2012::17:09:23 ===
Could not find exchange type x-federation.

=INFO REPORT==== 21-Aug-2012::17:09:23 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 21-Aug-2012::17:09:23 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 21-Aug-2012::17:09:23 ===
Could not find exchange type x-federation.

=INFO REPORT==== 21-Aug-2012::17:09:23 ===
started TCP Listener on 0.0.0.0:5672

=INFO REPORT==== 21-Aug-2012::17:09:24 ===
accepting AMQP connection <0.212.0> (192.168.78.18:55219 ->
192.168.78.16:5672)

=INFO REPORT==== 21-Aug-2012::17:09:24 ===
accepting AMQP connection <0.219.0> (192.168.78.17:58709 ->
192.168.78.16:5672)

=ERROR REPORT==== 21-Aug-2012::17:09:24 ===
connection <0.212.0>, channel 2 - error:
{amqp_error,not_found,
            "no exchange 'federation: skytap ->
rabbit at mq3.mpietrek.internal.illumita.com B' in vhost '/'",
            'exchange.delete'}

=INFO REPORT==== 21-Aug-2012::17:09:24 ===
accepting AMQP connection <0.233.0> (192.168.78.18:50211 ->
192.168.78.16:5672)

=ERROR REPORT==== 21-Aug-2012::17:09:24 ===
connection <0.219.0>, channel 2 - error:
{amqp_error,not_found,
            "no exchange 'federation: skytap ->
rabbit at mq2.mpietrek.internal.illumita.com A' in vhost '/'",
            'exchange.delete'}

=ERROR REPORT==== 21-Aug-2012::17:09:24 ===
connection <0.233.0>, channel 1 - error:
{amqp_error,command_invalid,"unknown exchange type 'x-federation-upstream'",
            'exchange.declare'}

=INFO REPORT==== 21-Aug-2012::17:09:24 ===
accepting AMQP connection <0.245.0> (192.168.78.17:58290 ->
192.168.78.16:5672)

=INFO REPORT==== 21-Aug-2012::17:09:24 ===
closing AMQP connection <0.233.0> (192.168.78.18:50211 -> 192.168.78.16:5672
)

=WARNING REPORT==== 21-Aug-2012::17:09:24 ===
Could not find exchange type x-federation.

=ERROR REPORT==== 21-Aug-2012::17:09:24 ===
connection <0.245.0>, channel 1 - error:
{amqp_error,command_invalid,"unknown exchange type 'x-federation-upstream'",
            'exchange.declare'}

=INFO REPORT==== 21-Aug-2012::17:09:24 ===
closing AMQP connection <0.245.0> (192.168.78.17:58290 -> 192.168.78.16:5672
)

=WARNING REPORT==== 21-Aug-2012::17:09:24 ===
Could not find exchange type x-federation.

----
And rabbit at mq2.log:

=WARNING REPORT==== 21-Aug-2012::17:09:20 ===
Connection (<0.300.0>) closing: received hard error {'connection.close',320,
                                                     <<"CONNECTION_FORCED -
broker forced connection closure with reason 'shutdown'">>,
                                                     0,0} from server

=INFO REPORT==== 21-Aug-2012::17:09:20 ===
Federation exchange 'skytap' in vhost '/' disconnected from mq1:/:skytap
{upstream_channel_down,
    {connection_closing,
        {server_initiated_close,320,
            <<"CONNECTION_FORCED - broker forced connection closure with
reason 'shutdown'">>}}}

=ERROR REPORT==== 21-Aug-2012::17:09:20 ===
** Generic server <0.300.0> terminating
** Last message in was {channel_exit,3,{writer,send_failed,{error,closed}}}
** When Server state == {state,amqp_network_connection,
                            {state,#Port<0.5842>,0,<0.304.0>,131072,
                                {server_initiated_close,320,
                                    <<"CONNECTION_FORCED - broker forced
connection closure with reason 'shutdown'">>},
                                true},
                            <0.299.0>,<0.302.0>,
                            {amqp_params_network,<<"guest">>,<<"guest">>,
                                <<"/">>,"mq1",5672,0,0,0,infinity,none,
                                [#Fun<amqp_auth_mechanisms.plain.3>,
                                 #Fun<amqp_auth_mechanisms.amqplain.3>],
                                [],[]},
                            0,
                            [{<<"capabilities">>,table,
                              [{<<"publisher_confirms">>,bool,true},
                               {<<"exchange_exchange_bindings">>,bool,true},
                               {<<"basic.nack">>,bool,true},
                               {<<"consumer_cancel_notify">>,bool,true}]},
                             {<<"copyright">>,longstr,
                              <<"Copyright (C) 2007-2012 VMware, Inc.">>},
                             {<<"information">>,longstr,
                              <<"Licensed under the MPL.  See
http://www.rabbitmq.com/">>},
                             {<<"platform">>,longstr,<<"Erlang/OTP">>},
                             {<<"product">>,longstr,<<"RabbitMQ">>},
                             {<<"version">>,longstr,<<"2.8.5">>}],
                            #Fun<amqp_connection_sup.0.39273983>,
                            #Fun<amqp_connection_sup.2.54430129>,
                            {closing,server_initiated_close,
                                {'connection.close',320,
                                    <<"CONNECTION_FORCED - broker forced
connection closure with reason 'shutdown'">>,
                                    0,0},
                                none}}
** Reason for termination ==
** {function_clause,
       [{amqp_network_connection,handle_message,
            [{channel_exit,3,{writer,send_failed,{error,closed}}},
             {state,#Port<0.5842>,0,<0.304.0>,131072,
                 {server_initiated_close,320,
                     <<"CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'">>},
                 true}],
            []},
        {amqp_gen_connection,callback,3,[]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,607}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

=ERROR REPORT==== 21-Aug-2012::17:09:20 ===
** Generic server <0.305.0> terminating
** Last message in was {inet_async,#Port<0.5842>,404,
                           {ok,<<0,10,0,50,1,64,75,67,79,78,78,69,67,84,73,
                                 79,78,95,70,79,82,67,69,68,32,45,32,98,114,
                                 111,107,101,114,32,102,111,114,99,101,100,

32,99,111,110,110,101,99,116,105,111,110,32,
                                 99,108,111,115,117,114,101,32,119,105,116,

104,32,114,101,97,115,111,110,32,39,115,104,
                                 117,116,100,111,119,110,39,0,0,0,0,206>>}}
** When Server state == {state,#Port<0.5842>,<0.300.0>,<0.302.0>,
                               {method,rabbit_framing_amqp_0_9_1},
                               none}
** Reason for termination ==
** {socket_error,einval}

=WARNING REPORT==== 21-Aug-2012::17:09:20 ===
Federation exchange 'skytap' in vhost '/' did not connect to mq1:/:skytap
{error,econnrefused}

=WARNING REPORT==== 21-Aug-2012::17:09:22 ===
Federation exchange 'skytap' in vhost '/' did not connect to mq1:/:skytap
{error,econnrefused}

=WARNING REPORT==== 21-Aug-2012::17:09:24 ===
Connection (<0.412.0>) closing: received hard error {'connection.close',503,
                                                     <<"COMMAND_INVALID -
unknown exchange type 'x-federation-upstream'">>,
                                                     40,10} from server

=INFO REPORT==== 21-Aug-2012::17:09:24 ===
Federation exchange 'skytap' in vhost '/' connected to mq1:/:skytap

---
The management console for mq2 shows this on the Exchanges tab:
federation: skytap -> rabbit at mq1.mpietrek.internal.illumita.com
A<http://mq2.mpietrek.internal.illumita.com:55672/#/exchanges/%2F/federation%3A%20skytap%20-%3E%20rabbit%40mq1.mpietrek.internal.illumita.com%20A>
x-federation-upstream D AD I Args

And mq3's Exchanges tab shows:
federation: skytap -> rabbit at mq1.mpietrek.internal.illumita.com
B<http://mq3.mpietrek.internal.illumita.com:55672/#/exchanges/%2F/federation%3A%20skytap%20-%3E%20rabbit%40mq1.mpietrek.internal.illumita.com%20B>
x-federation-upstream D AD I Args

Ideas on what I'm doing wrong, or if this is a known issue?

Thanks,

Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120821/affa88b6/attachment.htm>