[rabbitmq-discuss] Understanding Federation

James Carr james.r.carr at gmail.com
Mon Feb 27 21:13:05 GMT 2012


FYI, here is the configuration I am using on the downstream server:

[
  {rabbitmq_federation,
    [ {exchanges, [[{exchange,     "usage"},
                    {virtual_host, "/"},
                    {type,         "topic"},
                    {durable,      true},
                    {auto_delete,  false},
                    {internal,     false},
                    {upstream_set, "remotes"}]
                  ]},
      {upstream_sets, [{"remotes", [[{connection,  "upstream-server"},
                                     {exchange,    "usage"},
                                     {max_hops,    2}]
                                   ]}
                      ]},
      {connections, [{"upstream-server", [{host,            "dc2rabbitmq1"},
                                          {protocol,        "amqp"},
                                          {port,            "5672"},
                                          {virtual_host,    "/"},
                                          {username,        "guest"},
                                          {password,        "guest"},
                                          {mechanism,       default},
                                          {prefetch_count,  1000},
                                          {reconnect_delay, 5},
                                          {heartbeat,       1},
                                          {queue_expires,   30000},
                                          {message_ttl,     10000}
                                          ]}
                      ]},
      {local_username, "guest"},
      {local_nodename, "rabbit"}
    ]
  }
].


It creates the local exchange on the downstream node just fine, but no
action on the upstream (dc2rabbitmq1). A look at the error logs
reveals the below messages, but I don't completely grok erlang yet. :S

=ERROR REPORT==== 27-Feb-2012::15:01:20 ===
** Generic server <0.1963.0> terminating
** Last message in was connect
** When Server state == {state,amqp_network_connection,
                            {state,undefined,undefined,undefined,undefined,
                                undefined,false},
                            <0.1962.0>,undefined,
                            {amqp_params_network,<<"guest">>,<<"guest">>,
                                <<"/">>,"dc2rabbitmq1","5672",0,0,1,infinity,
                                none,
                                [#Fun<amqp_auth_mechanisms.plain.3>,
                                 #Fun<amqp_auth_mechanisms.amqplain.3>],
                                [],[]},
                            undefined,undefined,
                            #Fun<amqp_connection_sup.0.39273983>,
                            #Fun<amqp_connection_sup.2.54430129>,false}
** Reason for termination ==
** {function_clause,[{inet_tcp,getserv,["5672"]},
                     {gen_tcp,connect1,4},
                     {gen_tcp,connect,4},
                     {amqp_network_connection,do_connect,5},
                     {amqp_gen_connection,handle_call,3},
                     {gen_server,handle_msg,5},
                     {proc_lib,init_p_do_apply,3}]}

=ERROR REPORT==== 27-Feb-2012::15:01:20 ===
** Generic server <0.1949.0> terminating
** Last message in was {'$gen_cast',maybe_go}
** When Server state == {not_started,
                            {{upstream,
                                 {amqp_params_network,<<"guest">>,
                                     <<"guest">>,<<"/">>,"dc2rabbitmq1",
                                     "5672",0,0,1,infinity,none,
                                     [#Fun<amqp_auth_mechanisms.plain.3>,
                                      #Fun<amqp_auth_mechanisms.amqplain.3>],
                                     [],[]},
                                 <<"usage">>,1000,2,5,none,10000,
                                 "upstream-server"},
                             {resource,<<"/">>,exchange,<<"usage">>}}}
** Reason for termination ==
** {{function_clause,[{inet_tcp,getserv,["5672"]},
                      {gen_tcp,connect1,4},
                      {gen_tcp,connect,4},
                      {amqp_network_connection,do_connect,5},
                      {amqp_gen_connection,handle_call,3},
                      {gen_server,handle_msg,5},
                      {proc_lib,init_p_do_apply,3}]},
    {gen_server,call,[<0.1963.0>,connect,infinity]}}

=WARNING REPORT==== 27-Feb-2012::15:01:20 ===
Channel (<0.1958.0>): Unregistering confirm handler <0.1949.0> because
it died. Reason: {{function_clause,

                   [{inet_tcp,

                     getserv,

                     ["5672"]},

                    {gen_tcp,

                     connect1,

                     4},

                    {gen_tcp,

                     connect,

                     4},

                    {amqp_network_connection,

                     do_connect,

                     5},

                    {amqp_gen_connection,

                     handle_call,

                     3},

                    {gen_server,

                     handle_msg,

                     5},

                    {proc_lib,

                     init_p_do_apply,

                     3}]},

                  {gen_server,

                   call,

                   [<0.1963.0>,

                    connect,

                    infinity]}}

Any help would be appreciated.

Thanks,
James


On Mon, Feb 27, 2012 at 2:12 PM, James Carr <james.r.carr at gmail.com> wrote:
> Sorry to hijack an existing thread, but I'm delving into using
> federation myself (after a 3 month break of even working with
> rabbitmq) and have a couple questions.
>
> 1) Does federation need to be installed on both upstream and downstream?
> 2) I have a configuration that matches what Denis has but it only
> creates the exchange on the downstream server and does nothing
> upstream (only difference between his and mine is the remote hostname,
> of course). What am I missing?
>
>
> Thanks,
> James
>
>
> On Thu, Feb 23, 2012 at 10:47 AM, Simon MacMullen <simon at rabbitmq.com> wrote:
>> On 23/02/12 16:36, MELIQUE Denis (MORPHO) wrote:
>>>
>>> Thank you Simon.
>>>
>>> This makes it much clearer for me.
>>> I was first surprised to see this queue always empty on my test bed,
>>> but the reason is (correct me if I'm wrong)
>>> that my system was not loaded enough, and consumers got messages as soon
>>> as they came in.
>>
>>
>> You're partly right. Your system was not loaded enough, but the messages in
>> that queue will be consumed by the downstream exchange, not (directly) by
>> your consumers.
>>
>>
>>> So every message consumed from the downstream is first written on disk.
>>
>>
>> No. The queue is durable, but that just determines whether it survives a
>> restart. Transient messages can still through it, and remain transient.
>>
>> Cheers, Simon
>>
>>
>>> This might be a disadvantage for performance.
>>> And gives an advantage to Shovel, as you can configure it with transient
>>> queues.
>>>
>>> Denis
>>>
>>> -----Original Message-----
>>> From: Simon MacMullen [mailto:simon at rabbitmq.com]
>>> Sent: Thursday, February 23, 2012 12:11 PM
>>> To: MELIQUE Denis (MORPHO)
>>> Cc: rabbitmq-discuss at lists.rabbitmq.com
>>> Subject: Re: [rabbitmq-discuss] Understanding Federation
>>>
>>> On 23/02/12 10:56, MELIQUE Denis (MORPHO) wrote:
>>>>
>>>> I’m doing some testing with federation plug-in to try to understand it.
>>>
>>>
>>> Cool!
>>>
>>> <snip>
>>>
>>>> What are these Exchange and Queues used for ?
>>>
>>>
>>> The queue is there to buffer messages that are published to the upstream
>>> exchange and have not yet been consumed from the downstream exchange.
>>>
>>> It is bound to the internal exchange with the union of all the bindings
>>> that are in use downstream, so we only transfer messages that are going
>>> to be of use downstream.
>>>
>>> The internal exchange is there so that whenever we (re)establish a
>>> federation link we can set up an exact set of bindings
>>> mostly-atomically, by creating a new internal exchange, creating all the
>>> bindings to it, then binding it to the upstream exchange.
>>>
>>>> Are they reserved for federation internal use ?
>>>
>>>
>>> Yes.
>>>
>>>> When do messages go into these queues ?
>>>
>>>
>>> As mentioned, the bindings are pushed upstream so messages from upstream
>>> go into these queues if they will be routed once they get downstream.
>>>
>>>> Do they have to be durable or is there a way to configure them as
>>>> transient ?
>>>
>>>
>>> At the moment they have to be durable. Is this a problem? The idea is
>>> that you can set the queue TTL if you do not want them to hang around
>>> forever.
>>>
>>>> I would be very interested by a brief description of how federation
>>>> works internally.
>>>
>>>
>>> Hopefully this answers your question.
>>>
>>> Cheers, Simon
>>>
>>
>>
>> --
>> Simon MacMullen
>> RabbitMQ, VMware
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


More information about the rabbitmq-discuss mailing list