[rabbitmq-discuss] rabbit federation is failing
Joao Gomes Pereira
joao.pereira at mobbit.com
Mon Aug 19 15:20:06 BST 2013
My computers are rebooting every night at 00:30, without stoping
rabbitMQ properly.
Could this be the problem? Because these errors in the federations show
up in the first messages in the morning:
=ERROR REPORT==== 19-Aug-2013::08:51:36 ===
** Generic server <0.287.0> terminating
** Last message in was heartbeat_timeout
** When Server state == {state,amqp_network_connection,
{state,#Port<0.12096>,600,<0.298.0>,131072,
undefined,false},
<0.285.0>,<0.289.0>,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"94.46.208.100",5672,0,0,0,infinity,
none,
[#Fun<amqp_uri.7.32597394>,
#Fun<amqp_uri.7.32597394>],
[],[]},
0,
[{<<"capabilities">>,table,
[{<<"publisher_confirms">>,bool,true},
{<<"exchange_exchange_bindings">>,bool,true},
{<<"basic.nack">>,bool,true},
{<<"consumer_cancel_notify">>,bool,true}]},
{<<"copyright">>,longstr,
<<"Copyright (C) 2007-2013 VMware, Inc.">>},
{<<"information">>,longstr,
<<"Licensed under the MPL. See
http://www.rabbitmq.com/">>},
{<<"platform">>,longstr,<<"Erlang/OTP">>},
{<<"product">>,longstr,<<"RabbitMQ">>},
{<<"version">>,longstr,<<"3.1.3">>}],
#Fun<amqp_connection_sup.0.39273983>,
#Fun<amqp_connection_sup.2.54430129>,false}
** Reason for termination ==
** heartbeat_timeout
=ERROR REPORT==== 19-Aug-2013::08:51:37 ===
** Generic server <0.234.0> terminating
** Last message in was {'DOWN',#Ref<0.0.0.1535>,process,<0.327.0>,shutdown}
** When Server state == {state,
{upstream,
[<<"amqp://94.46.208.100">>],
<<"LX">>,1000,1,1,36000000,none,false,'on-confirm',
none,<<"my-upstream">>},
{upstream_params,<<"amqp://94.46.208.100">>,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"94.46.208.100",undefined,0,0,0,infinity,
none,
[#Fun<amqp_uri.7.32597394>,
#Fun<amqp_uri.7.32597394>],
[],[]},
{exchange,
{resource,<<"/">>,exchange,<<"LX">>},
fanout,true,false,false,[],
[{federation,
[{{<<"my-upstream">>,<<"LX">>},<<"B">>}]}],
[{vhost,<<"/">>},
{name,<<"federate-me">>},
{pattern,<<"LX">>},
{definition,
[{<<"federation-upstream-set">>,<<"all">>}]},
{priority,0}],
{[],[rabbit_federation_exchange]}}},
<0.287.0>,<0.327.0>,
<<"amq.ctag-48dZ1r0vSUQ_DIeVzVnH3g">>,
<<"federation: LX -> rabbit at europcar-disp-lx">>,
<<"federation: LX -> rabbit at europcar-disp-lx A">>,
{0,nil},
2,
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],
[[{<<>>,[]}|
{set,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},
{{[],[],[],[],[],[],[],[],
[{resource,<<"/">>,queue,
<<"amq.gen-Lo9ghzM3Z7DLjpS_QXYJMw">>}],
[],[],[],[],[],[],[]}}}]],
[],[]}}},
<0.259.0>,<0.278.0>,
{resource,<<"/">>,exchange,<<"LX">>},
{0,nil}}
** Reason for termination ==
** {upstream_channel_down,shutdown}
On 08/19/2013 02:30 PM, Joao Gomes Pereira wrote:
> They both failed, the upstream and the downstream.
> The status was showing "running" in both sites. I didnt used the
> command "rabbitmqctl"
> but i checked in the RabbitMQ managment (web interface).
>
> Is this timeout normal? how can we deal with this situation in order
> to have the nodes federated 24/7 ?
>
> Thanks
> Joao Pereira
>
>
>
> On 08/15/2013 12:31 PM, Simon MacMullen wrote:
>> It's not obvious that those errors have anything to do with your
>> problem.
>>
>> The first one is an inbound connection being dropped due to its
>> heartbeat being missed.
>>
>> The second is an error which (unfortunately) happens whenever a
>> federation link comes up; it needs to ensure that an exchange does
>> not exist, so it tries to delete it - if it already did not exist we
>> log an error.
>>
>> Federation should always re-establish links that have failed.
>>
>> So when you say it stopped working, what happened? Is 192.168.1.200
>> the upstream, downstream or both? What does the link status reporting
>> say? (http://www.rabbitmq.com/federation.html#status)
>>
>> Cheers, Simon
>>
>> On 14/08/13 17:52, Joao Gomes Pereira wrote:
>>> Hello
>>> I implemented a rabbit federation to exchange SMS messages between
>>> diferent servers.
>>> After working fine for some days, one of my nodes (192.168.1.200) just
>>> stopped working.
>>>
>>> We can see some errors in the node log:
>>>
>>>
>>> =INFO REPORT==== 13-Aug-2013::14:01:31 ===
>>> Federation exchange 'LX' in vhost '/' connected to exchange 'LX' in
>>> vhost '/' on amqp://94.46.208.100
>>>
>>> =ERROR REPORT==== 13-Aug-2013::14:03:46 ===
>>> closing AMQP connection <0.5257.0> (94.46.208.98:45341 ->
>>> 192.168.1.200:5672):
>>> {heartbeat_timeout,running}
>>>
>>> =INFO REPORT==== 13-Aug-2013::14:09:27 ===
>>> accepting AMQP connection <0.5960.0> (94.46.208.98:60137 ->
>>> 192.168.1.200:5672)
>>>
>>> =ERROR REPORT==== 13-Aug-2013::14:09:27 ===
>>> connection <0.5960.0>, channel 3 - soft error:
>>> {amqp_error,not_found,
>>> "no exchange 'federation: SMS_System ->
>>> SMSDriver+Dispatcher-rabbit at insight-DISP-rabbit02 B' in vhost '/'",
>>> 'exchange.delete'}
>>>
>>> =INFO REPORT==== 13-Aug-2013::14:09:27 ===
>>> accepting AMQP connection <0.5978.0> (94.46.208.98:20734 ->
>>> 192.168.1.200:5672)
>>>
>>> =INFO REPORT==== 13-Aug-2013::14:09:27 ===
>>> closing AMQP connection <0.5978.0> (94.46.208.98:20734 ->
>>> 192.168.1.200:5672)
>>>
>>>
>>>
>>>
>>> How can I configure my node to be always federated?
>>>
>>> Thanks
>>> regards
>>> Joao Pereira
>>>
>>>
>>>
>>
>>
>
>
--
More information about the rabbitmq-discuss
mailing list