I've got a head scratcher here with regards to federation.<br><br>We have two brokers running 2.8.6. I want to establish bidirectional federated topic exchanges between the brokers. However, when I start up all the brokers the federation tab in the Web UI shows "starting" indefinitely.<br>
<br>Looking at the logs, I see a stream of messages like this from each broker:<br><br><<< from host util1 >>><br>=WARNING REPORT==== 14-Nov-2012::11:49:19 ===<br>Federation exchange 'skytap' in vhost '/' did not connect to west1util1:55672:/:skytap<br>
{error,handshake_receive_timed_out}<br><br><br><<< from host west1util1 >>><br>=WARNING REPORT==== 14-Nov-2012::10:41:41 ===<br>Federation exchange 'skytap' in vhost '/' did not connect to util1:55672:/:skytap<br>
{error,handshake_receive_timed_out}<br><br><br>Using Wireshark, I've verified that traffic on port 55672 is going back and forth between the two machines, corresponding to the 1 minute connection retry intervals. See very end of this message for packet summary.<br>
<br>Both brokers appear otherwise healthy and the "Exchanges" tab shows "skytap x-federation D Args" on both brokers.<br><br>This exact same setup has worked previously on several different testbeds. My rabbitmq.config file is generated programmatically and has worked reliably in other federation scenarios, so I'm reasonably sure it's not just a fat-fingered rabbitmq.config file. That said, the config file looks like this:<br>
<br>[<br>{rabbit, [{cluster_nodes, [rabbit@util1]}, {disk_free_limit, 104857600}]},<br>{mnesia, [{debug, none}]},<br>{rabbitmq_federation, [{exchanges, [[{type, "topic"}, {upstream_set, "skytap_upstreams"}, {virtual_host, "/"}, {exchange, "skytap"}]]}, <br>
{upstream_sets, [{connection, "upstream-abc123-cloud/west1"}, {max_hops, 1}, {exchange, "skytap"}]]}]}, <br>{connections, [{"upstream-abc123-cloud/west1", [{username, "foo"}, {host, "west1util1"}, {password, "foo"}, {port, 55672}]}]}]}<br>
<br>I don't see anything else interesting in the other log files. Any hints on how I can track this down?<br><br>Thanks,<br><br>Matt<br><br>Packet summary:<br><br>No.���� Time������� Source��������������� Destination���������� Protocol Info<br>
����� 1 0.000000��� 10.1.208.5����������� 10.1.208.39���������� TCP����� 40729 > 55672 [FIN, ACK] Seq=1 Ack=1 Win=92 Len=0 TSV=466020877 TSER=354966603<br>����� 2 0.001593��� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55672 > 40729 [FIN, ACK] Seq=1 Ack=2 Win=91 Len=0 TSV=354972603 TSER=466020877<br>
����� 3 0.001607��� 10.1.208.5����������� 10.1.208.39���������� TCP����� 40729 > 55672 [ACK] Seq=2 Ack=2 Win=92 Len=0 TSV=466020878 TSER=354972603<br>����� 4 0.020673��� 10.1.208.5����������� 10.1.208.39���������� TCP����� 42451 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466020880 TSER=0 WS=6<br>
����� 5 0.022202��� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55672 > 42451 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354972605 TSER=466020880 WS=6<br>����� 6 0.022215��� 10.1.208.5����������� 10.1.208.39���������� TCP����� 42451 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466020880 TSER=354972605<br>
����� 7 0.022280��� 10.1.208.5����������� 10.1.208.39���������� TCP����� 42451 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466020880 TSER=354972605<br>����� 8 0.023274��� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55672 > 42451 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354972605 TSER=466020880<br>
���� 14 57.211788�� 10.1.208.39���������� 10.1.208.5����������� TCP����� 56169 > 55672 [FIN, ACK] Seq=1 Ack=1 Win=92 Len=0 TSV=354978324 TSER=466020599<br>���� 15 57.211854�� 10.1.208.5����������� 10.1.208.39���������� TCP����� 55672 > 56169 [FIN, ACK] Seq=1 Ack=2 Win=91 Len=0 TSV=466026599 TSER=354978324<br>
���� 16 57.212561�� 10.1.208.39���������� 10.1.208.5����������� TCP����� 56169 > 55672 [ACK] Seq=2 Ack=2 Win=92 Len=0 TSV=354978324 TSER=466026599<br>���� 17 57.231459�� 10.1.208.39���������� 10.1.208.5����������� TCP����� 33070 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=354978326 TSER=0 WS=6<br>
���� 18 57.231491�� 10.1.208.5����������� 10.1.208.39���������� TCP����� 55672 > 33070 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=466026601 TSER=354978326 WS=6<br>���� 19 57.232238�� 10.1.208.39���������� 10.1.208.5����������� TCP����� 33070 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=354978326 TSER=466026601<br>
���� 20 57.232266�� 10.1.208.39���������� 10.1.208.5����������� TCP����� 33070 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=354978326 TSER=466026601<br>���� 21 57.232273�� 10.1.208.5����������� 10.1.208.39���������� TCP����� 55672 > 33070 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=466026601 TSER=354978326<br>
���� 22 60.023027�� 10.1.208.5����������� 10.1.208.39���������� TCP����� 42451 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=466026880 TSER=354972605<br>���� 23 60.024243�� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55672 > 42451 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=354978605 TSER=466026880<br>
���� 24 60.024259�� 10.1.208.5����������� 10.1.208.39���������� TCP����� 42451 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=466026880 TSER=354978605<br>���� 25 60.054155�� 10.1.208.5����������� 10.1.208.39���������� TCP����� 43254 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466026883 TSER=0 WS=6<br>
���� 26 60.055619�� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55672 > 43254 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354978608 TSER=466026883 WS=6<br>���� 27 60.055637�� 10.1.208.5����������� 10.1.208.39���������� TCP����� 43254 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466026883 TSER=354978608<br>
���� 28 60.055699�� 10.1.208.5����������� 10.1.208.39���������� TCP����� 43254 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466026883 TSER=354978608<br>���� 29 60.056525�� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55672 > 43254 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354978608 TSER=466026883<br>
���� 35 117.235482� 10.1.208.39���������� 10.1.208.5����������� TCP����� 33070 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=354984326 TSER=466026601<br>���� 36 117.235554� 10.1.208.5����������� 10.1.208.39���������� TCP����� 55672 > 33070 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=466032601 TSER=354984326<br>
���� 37 117.236376� 10.1.208.39���������� 10.1.208.5����������� TCP����� 33070 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=354984326 TSER=466032601<br>���� 38 117.257662� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55930 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=354984329 TSER=0 WS=6<br>
���� 39 117.257693� 10.1.208.5����������� 10.1.208.39���������� TCP����� 55672 > 55930 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=466032603 TSER=354984329 WS=6<br>���� 40 117.258899� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55930 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=354984329 TSER=466032603<br>
���� 41 117.258924� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55930 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=354984329 TSER=466032603<br>���� 42 117.258931� 10.1.208.5����������� 10.1.208.39���������� TCP����� 55672 > 55930 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=466032603 TSER=354984329<br>
���� 43 120.056005� 10.1.208.5����������� 10.1.208.39���������� TCP����� 43254 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=466032883 TSER=354978608<br>���� 44 120.057752� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55672 > 43254 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=354984609 TSER=466032883<br>
���� 45 120.057781� 10.1.208.5����������� 10.1.208.39���������� TCP����� 43254 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=466032883 TSER=354984609<br>���� 46 120.077268� 10.1.208.5����������� 10.1.208.39���������� TCP����� 33297 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466032885 TSER=0 WS=6<br>
���� 47 120.079009� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55672 > 33297 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354984611 TSER=466032885 WS=6<br>���� 48 120.079030� 10.1.208.5����������� 10.1.208.39���������� TCP����� 33297 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466032885 TSER=354984611<br>
���� 49 120.079096� 10.1.208.5����������� 10.1.208.39���������� TCP����� 33297 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466032885 TSER=354984611<br>���� 50 120.079884� 10.1.208.39���������� 10.1.208.5����������� TCP����� 55672 > 33297 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354984611 TSER=466032885<br>