[rabbitmq-discuss] rabbit federation is failing

Mon Aug 19 15:20:06 BST 2013

My computers are rebooting every night at 00:30, without stoping 
rabbitMQ properly.
Could this be the problem? Because these errors in the federations show 
up in the first messages in the morning:

=ERROR REPORT==== 19-Aug-2013::08:51:36 ===
** Generic server <0.287.0> terminating
** Last message in was heartbeat_timeout
** When Server state == {state,amqp_network_connection,
{state,#Port<0.12096>,600,<0.298.0>,131072,
                                 undefined,false},
                             <0.285.0>,<0.289.0>,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"94.46.208.100",5672,0,0,0,infinity,
                                 none,
                                 [#Fun<amqp_uri.7.32597394>,
                                  #Fun<amqp_uri.7.32597394>],
                                 [],[]},
                             0,
[{<<"capabilities">>,table,
[{<<"publisher_confirms">>,bool,true},
{<<"exchange_exchange_bindings">>,bool,true},
{<<"basic.nack">>,bool,true},
{<<"consumer_cancel_notify">>,bool,true}]},
                              {<<"copyright">>,longstr,
                               <<"Copyright (C) 2007-2013 VMware, Inc.">>},
{<<"information">>,longstr,
                               <<"Licensed under the MPL. See 
http://www.rabbitmq.com/">>},
{<<"platform">>,longstr,<<"Erlang/OTP">>},
{<<"product">>,longstr,<<"RabbitMQ">>},
{<<"version">>,longstr,<<"3.1.3">>}],
#Fun<amqp_connection_sup.0.39273983>,
#Fun<amqp_connection_sup.2.54430129>,false}
** Reason for termination ==
** heartbeat_timeout

=ERROR REPORT==== 19-Aug-2013::08:51:37 ===
** Generic server <0.234.0> terminating
** Last message in was {'DOWN',#Ref<0.0.0.1535>,process,<0.327.0>,shutdown}
** When Server state == {state,
                          {upstream,
[<<"amqp://94.46.208.100">>],
<<"LX">>,1000,1,1,36000000,none,false,'on-confirm',
                           none,<<"my-upstream">>},
{upstream_params,<<"amqp://94.46.208.100">>,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"94.46.208.100",undefined,0,0,0,infinity,
                            none,
                            [#Fun<amqp_uri.7.32597394>,
                             #Fun<amqp_uri.7.32597394>],
                            [],[]},
                           {exchange,
{resource,<<"/">>,exchange,<<"LX">>},
                            fanout,true,false,false,[],
                            [{federation,
[{{<<"my-upstream">>,<<"LX">>},<<"B">>}]}],
                            [{vhost,<<"/">>},
                             {name,<<"federate-me">>},
                             {pattern,<<"LX">>},
                             {definition,
[{<<"federation-upstream-set">>,<<"all">>}]},
                             {priority,0}],
                            {[],[rabbit_federation_exchange]}}},
                          <0.287.0>,<0.327.0>,
<<"amq.ctag-48dZ1r0vSUQ_DIeVzVnH3g">>,
                          <<"federation: LX -> rabbit at europcar-disp-lx">>,
                          <<"federation: LX -> rabbit at europcar-disp-lx A">>,
                          {0,nil},
                          2,
                          {dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],
                             [[{<<>>,[]}|
                               {set,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],
[{resource,<<"/">>,queue,
<<"amq.gen-Lo9ghzM3Z7DLjpS_QXYJMw">>}],
                                  [],[],[],[],[],[],[]}}}]],
                             [],[]}}},
                          <0.259.0>,<0.278.0>,
{resource,<<"/">>,exchange,<<"LX">>},
                          {0,nil}}
** Reason for termination ==
** {upstream_channel_down,shutdown}

On 08/19/2013 02:30 PM, Joao Gomes Pereira wrote:
> They both failed, the upstream and the downstream.
> The status was showing "running" in both sites. I didnt used the 
> command "rabbitmqctl"
> but i checked in the RabbitMQ managment (web interface).
>
> Is this timeout normal? how can we deal with this situation in order 
> to have the nodes federated 24/7 ?
>
> Thanks
> Joao Pereira
>
>
>
> On 08/15/2013 12:31 PM, Simon MacMullen wrote:
>> It's not obvious that those errors have anything to do with your 
>> problem.
>>
>> The first one is an inbound connection being dropped due to its 
>> heartbeat being missed.
>>
>> The second is an error which (unfortunately) happens whenever a 
>> federation link comes up; it needs to ensure that an exchange does 
>> not exist, so it tries to delete it - if it already did not exist we 
>> log an error.
>>
>> Federation should always re-establish links that have failed.
>>
>> So when you say it stopped working, what happened? Is 192.168.1.200 
>> the upstream, downstream or both? What does the link status reporting 
>> say? (http://www.rabbitmq.com/federation.html#status)
>>
>> Cheers, Simon
>>
>> On 14/08/13 17:52, Joao Gomes Pereira wrote:
>>> Hello
>>> I implemented a rabbit federation to exchange SMS messages between
>>> diferent servers.
>>> After working fine for some days, one of my nodes (192.168.1.200) just
>>> stopped working.
>>>
>>> We can see some errors in the node log:
>>>
>>>
>>> =INFO REPORT==== 13-Aug-2013::14:01:31 ===
>>> Federation exchange 'LX' in vhost '/' connected to exchange 'LX' in
>>> vhost '/' on amqp://94.46.208.100
>>>
>>> =ERROR REPORT==== 13-Aug-2013::14:03:46 ===
>>> closing AMQP connection <0.5257.0> (94.46.208.98:45341 ->
>>> 192.168.1.200:5672):
>>> {heartbeat_timeout,running}
>>>
>>> =INFO REPORT==== 13-Aug-2013::14:09:27 ===
>>> accepting AMQP connection <0.5960.0> (94.46.208.98:60137 ->
>>> 192.168.1.200:5672)
>>>
>>> =ERROR REPORT==== 13-Aug-2013::14:09:27 ===
>>> connection <0.5960.0>, channel 3 - soft error:
>>> {amqp_error,not_found,
>>>              "no exchange 'federation: SMS_System ->
>>> SMSDriver+Dispatcher-rabbit at insight-DISP-rabbit02 B' in vhost '/'",
>>>              'exchange.delete'}
>>>
>>> =INFO REPORT==== 13-Aug-2013::14:09:27 ===
>>> accepting AMQP connection <0.5978.0> (94.46.208.98:20734 ->
>>> 192.168.1.200:5672)
>>>
>>> =INFO REPORT==== 13-Aug-2013::14:09:27 ===
>>> closing AMQP connection <0.5978.0> (94.46.208.98:20734 ->
>>> 192.168.1.200:5672)
>>>
>>>
>>>
>>>
>>> How can I configure my node to be always federated?
>>>
>>> Thanks
>>> regards
>>> Joao Pereira
>>>
>>>
>>>
>>
>>
>
>

-- 