[rabbitmq-discuss] Understanding Federation

James Carr james.r.carr at gmail.com
Mon Feb 27 21:48:22 GMT 2012


Nvm, I got it working.

On Mon, Feb 27, 2012 at 3:13 PM, James Carr <james.r.carr at gmail.com> wrote:
> FYI, here is the configuration I am using on the downstream server:
>
> [
>  {rabbitmq_federation,
>    [ {exchanges, [[{exchange,     "usage"},
>                    {virtual_host, "/"},
>                    {type,         "topic"},
>                    {durable,      true},
>                    {auto_delete,  false},
>                    {internal,     false},
>                    {upstream_set, "remotes"}]
>                  ]},
>      {upstream_sets, [{"remotes", [[{connection,  "upstream-server"},
>                                     {exchange,    "usage"},
>                                     {max_hops,    2}]
>                                   ]}
>                      ]},
>      {connections, [{"upstream-server", [{host,            "dc2rabbitmq1"},
>                                          {protocol,        "amqp"},
>                                          {port,            "5672"},
>                                          {virtual_host,    "/"},
>                                          {username,        "guest"},
>                                          {password,        "guest"},
>                                          {mechanism,       default},
>                                          {prefetch_count,  1000},
>                                          {reconnect_delay, 5},
>                                          {heartbeat,       1},
>                                          {queue_expires,   30000},
>                                          {message_ttl,     10000}
>                                          ]}
>                      ]},
>      {local_username, "guest"},
>      {local_nodename, "rabbit"}
>    ]
>  }
> ].
>
>
> It creates the local exchange on the downstream node just fine, but no
> action on the upstream (dc2rabbitmq1). A look at the error logs
> reveals the below messages, but I don't completely grok erlang yet. :S
>
> =ERROR REPORT==== 27-Feb-2012::15:01:20 ===
> ** Generic server <0.1963.0> terminating
> ** Last message in was connect
> ** When Server state == {state,amqp_network_connection,
>                            {state,undefined,undefined,undefined,undefined,
>                                undefined,false},
>                            <0.1962.0>,undefined,
>                            {amqp_params_network,<<"guest">>,<<"guest">>,
>                                <<"/">>,"dc2rabbitmq1","5672",0,0,1,infinity,
>                                none,
>                                [#Fun<amqp_auth_mechanisms.plain.3>,
>                                 #Fun<amqp_auth_mechanisms.amqplain.3>],
>                                [],[]},
>                            undefined,undefined,
>                            #Fun<amqp_connection_sup.0.39273983>,
>                            #Fun<amqp_connection_sup.2.54430129>,false}
> ** Reason for termination ==
> ** {function_clause,[{inet_tcp,getserv,["5672"]},
>                     {gen_tcp,connect1,4},
>                     {gen_tcp,connect,4},
>                     {amqp_network_connection,do_connect,5},
>                     {amqp_gen_connection,handle_call,3},
>                     {gen_server,handle_msg,5},
>                     {proc_lib,init_p_do_apply,3}]}
>
> =ERROR REPORT==== 27-Feb-2012::15:01:20 ===
> ** Generic server <0.1949.0> terminating
> ** Last message in was {'$gen_cast',maybe_go}
> ** When Server state == {not_started,
>                            {{upstream,
>                                 {amqp_params_network,<<"guest">>,
>                                     <<"guest">>,<<"/">>,"dc2rabbitmq1",
>                                     "5672",0,0,1,infinity,none,
>                                     [#Fun<amqp_auth_mechanisms.plain.3>,
>                                      #Fun<amqp_auth_mechanisms.amqplain.3>],
>                                     [],[]},
>                                 <<"usage">>,1000,2,5,none,10000,
>                                 "upstream-server"},
>                             {resource,<<"/">>,exchange,<<"usage">>}}}
> ** Reason for termination ==
> ** {{function_clause,[{inet_tcp,getserv,["5672"]},
>                      {gen_tcp,connect1,4},
>                      {gen_tcp,connect,4},
>                      {amqp_network_connection,do_connect,5},
>                      {amqp_gen_connection,handle_call,3},
>                      {gen_server,handle_msg,5},
>                      {proc_lib,init_p_do_apply,3}]},
>    {gen_server,call,[<0.1963.0>,connect,infinity]}}
>
> =WARNING REPORT==== 27-Feb-2012::15:01:20 ===
> Channel (<0.1958.0>): Unregistering confirm handler <0.1949.0> because
> it died. Reason: {{function_clause,
>
>                   [{inet_tcp,
>
>                     getserv,
>
>                     ["5672"]},
>
>                    {gen_tcp,
>
>                     connect1,
>
>                     4},
>
>                    {gen_tcp,
>
>                     connect,
>
>                     4},
>
>                    {amqp_network_connection,
>
>                     do_connect,
>
>                     5},
>
>                    {amqp_gen_connection,
>
>                     handle_call,
>
>                     3},
>
>                    {gen_server,
>
>                     handle_msg,
>
>                     5},
>
>                    {proc_lib,
>
>                     init_p_do_apply,
>
>                     3}]},
>
>                  {gen_server,
>
>                   call,
>
>                   [<0.1963.0>,
>
>                    connect,
>
>                    infinity]}}
>
> Any help would be appreciated.
>
> Thanks,
> James
>
>
> On Mon, Feb 27, 2012 at 2:12 PM, James Carr <james.r.carr at gmail.com> wrote:
>> Sorry to hijack an existing thread, but I'm delving into using
>> federation myself (after a 3 month break of even working with
>> rabbitmq) and have a couple questions.
>>
>> 1) Does federation need to be installed on both upstream and downstream?
>> 2) I have a configuration that matches what Denis has but it only
>> creates the exchange on the downstream server and does nothing
>> upstream (only difference between his and mine is the remote hostname,
>> of course). What am I missing?
>>
>>
>> Thanks,
>> James
>>
>>
>> On Thu, Feb 23, 2012 at 10:47 AM, Simon MacMullen <simon at rabbitmq.com> wrote:
>>> On 23/02/12 16:36, MELIQUE Denis (MORPHO) wrote:
>>>>
>>>> Thank you Simon.
>>>>
>>>> This makes it much clearer for me.
>>>> I was first surprised to see this queue always empty on my test bed,
>>>> but the reason is (correct me if I'm wrong)
>>>> that my system was not loaded enough, and consumers got messages as soon
>>>> as they came in.
>>>
>>>
>>> You're partly right. Your system was not loaded enough, but the messages in
>>> that queue will be consumed by the downstream exchange, not (directly) by
>>> your consumers.
>>>
>>>
>>>> So every message consumed from the downstream is first written on disk.
>>>
>>>
>>> No. The queue is durable, but that just determines whether it survives a
>>> restart. Transient messages can still through it, and remain transient.
>>>
>>> Cheers, Simon
>>>
>>>
>>>> This might be a disadvantage for performance.
>>>> And gives an advantage to Shovel, as you can configure it with transient
>>>> queues.
>>>>
>>>> Denis
>>>>
>>>> -----Original Message-----
>>>> From: Simon MacMullen [mailto:simon at rabbitmq.com]
>>>> Sent: Thursday, February 23, 2012 12:11 PM
>>>> To: MELIQUE Denis (MORPHO)
>>>> Cc: rabbitmq-discuss at lists.rabbitmq.com
>>>> Subject: Re: [rabbitmq-discuss] Understanding Federation
>>>>
>>>> On 23/02/12 10:56, MELIQUE Denis (MORPHO) wrote:
>>>>>
>>>>> I’m doing some testing with federation plug-in to try to understand it.
>>>>
>>>>
>>>> Cool!
>>>>
>>>> <snip>
>>>>
>>>>> What are these Exchange and Queues used for ?
>>>>
>>>>
>>>> The queue is there to buffer messages that are published to the upstream
>>>> exchange and have not yet been consumed from the downstream exchange.
>>>>
>>>> It is bound to the internal exchange with the union of all the bindings
>>>> that are in use downstream, so we only transfer messages that are going
>>>> to be of use downstream.
>>>>
>>>> The internal exchange is there so that whenever we (re)establish a
>>>> federation link we can set up an exact set of bindings
>>>> mostly-atomically, by creating a new internal exchange, creating all the
>>>> bindings to it, then binding it to the upstream exchange.
>>>>
>>>>> Are they reserved for federation internal use ?
>>>>
>>>>
>>>> Yes.
>>>>
>>>>> When do messages go into these queues ?
>>>>
>>>>
>>>> As mentioned, the bindings are pushed upstream so messages from upstream
>>>> go into these queues if they will be routed once they get downstream.
>>>>
>>>>> Do they have to be durable or is there a way to configure them as
>>>>> transient ?
>>>>
>>>>
>>>> At the moment they have to be durable. Is this a problem? The idea is
>>>> that you can set the queue TTL if you do not want them to hang around
>>>> forever.
>>>>
>>>>> I would be very interested by a brief description of how federation
>>>>> works internally.
>>>>
>>>>
>>>> Hopefully this answers your question.
>>>>
>>>> Cheers, Simon
>>>>
>>>
>>>
>>> --
>>> Simon MacMullen
>>> RabbitMQ, VMware
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


More information about the rabbitmq-discuss mailing list