I've got a head scratcher here with regards to federation.<br><br>We have two brokers running 2.8.6. I want to establish bidirectional federated topic exchanges between the brokers. However, when I start up all the brokers the federation tab in the Web UI shows "starting" indefinitely.<br>
<br>Looking at the logs, I see a stream of messages like this from each broker:<br><br><<< from host util1 >>><br>=WARNING REPORT==== 14-Nov-2012::11:49:19 ===<br>Federation exchange 'skytap' in vhost '/' did not connect to west1util1:55672:/:skytap<br>
{error,handshake_receive_timed_out}<br><br><br><<< from host west1util1 >>><br>=WARNING REPORT==== 14-Nov-2012::10:41:41 ===<br>Federation exchange 'skytap' in vhost '/' did not connect to util1:55672:/:skytap<br>
{error,handshake_receive_timed_out}<br><br><br>Using Wireshark, I've verified that traffic on port 55672 is going back and forth between the two machines, corresponding to the 1 minute connection retry intervals. See very end of this message for packet summary.<br>
<br>Both brokers appear otherwise healthy and the "Exchanges" tab shows "skytap x-federation D Args" on both brokers.<br><br>This exact same setup has worked previously on several different testbeds. My rabbitmq.config file is generated programmatically and has worked reliably in other federation scenarios, so I'm reasonably sure it's not just a fat-fingered rabbitmq.config file. That said, the config file looks like this:<br>
<br>[<br>{rabbit, [{cluster_nodes, [rabbit@util1]}, {disk_free_limit, 104857600}]},<br>{mnesia, [{debug, none}]},<br>{rabbitmq_federation, [{exchanges, [[{type, "topic"}, {upstream_set, "skytap_upstreams"}, {virtual_host, "/"}, {exchange, "skytap"}]]}, <br>
{upstream_sets, [{connection, "upstream-abc123-cloud/west1"}, {max_hops, 1}, {exchange, "skytap"}]]}]}, <br>{connections, [{"upstream-abc123-cloud/west1", [{username, "foo"}, {host, "west1util1"}, {password, "foo"}, {port, 55672}]}]}]}<br>
<br>I don't see anything else interesting in the other log files. Any hints on how I can track this down?<br><br>Thanks,<br><br>Matt<br><br>Packet summary:<br><br>No. Time Source Destination Protocol Info<br>
1 0.000000 10.1.208.5 10.1.208.39 TCP 40729 > 55672 [FIN, ACK] Seq=1 Ack=1 Win=92 Len=0 TSV=466020877 TSER=354966603<br> 2 0.001593 10.1.208.39 10.1.208.5 TCP 55672 > 40729 [FIN, ACK] Seq=1 Ack=2 Win=91 Len=0 TSV=354972603 TSER=466020877<br>
3 0.001607 10.1.208.5 10.1.208.39 TCP 40729 > 55672 [ACK] Seq=2 Ack=2 Win=92 Len=0 TSV=466020878 TSER=354972603<br> 4 0.020673 10.1.208.5 10.1.208.39 TCP 42451 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466020880 TSER=0 WS=6<br>
5 0.022202 10.1.208.39 10.1.208.5 TCP 55672 > 42451 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354972605 TSER=466020880 WS=6<br> 6 0.022215 10.1.208.5 10.1.208.39 TCP 42451 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466020880 TSER=354972605<br>
7 0.022280 10.1.208.5 10.1.208.39 TCP 42451 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466020880 TSER=354972605<br> 8 0.023274 10.1.208.39 10.1.208.5 TCP 55672 > 42451 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354972605 TSER=466020880<br>
14 57.211788 10.1.208.39 10.1.208.5 TCP 56169 > 55672 [FIN, ACK] Seq=1 Ack=1 Win=92 Len=0 TSV=354978324 TSER=466020599<br> 15 57.211854 10.1.208.5 10.1.208.39 TCP 55672 > 56169 [FIN, ACK] Seq=1 Ack=2 Win=91 Len=0 TSV=466026599 TSER=354978324<br>
16 57.212561 10.1.208.39 10.1.208.5 TCP 56169 > 55672 [ACK] Seq=2 Ack=2 Win=92 Len=0 TSV=354978324 TSER=466026599<br> 17 57.231459 10.1.208.39 10.1.208.5 TCP 33070 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=354978326 TSER=0 WS=6<br>
18 57.231491 10.1.208.5 10.1.208.39 TCP 55672 > 33070 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=466026601 TSER=354978326 WS=6<br> 19 57.232238 10.1.208.39 10.1.208.5 TCP 33070 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=354978326 TSER=466026601<br>
20 57.232266 10.1.208.39 10.1.208.5 TCP 33070 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=354978326 TSER=466026601<br> 21 57.232273 10.1.208.5 10.1.208.39 TCP 55672 > 33070 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=466026601 TSER=354978326<br>
22 60.023027 10.1.208.5 10.1.208.39 TCP 42451 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=466026880 TSER=354972605<br> 23 60.024243 10.1.208.39 10.1.208.5 TCP 55672 > 42451 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=354978605 TSER=466026880<br>
24 60.024259 10.1.208.5 10.1.208.39 TCP 42451 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=466026880 TSER=354978605<br> 25 60.054155 10.1.208.5 10.1.208.39 TCP 43254 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466026883 TSER=0 WS=6<br>
26 60.055619 10.1.208.39 10.1.208.5 TCP 55672 > 43254 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354978608 TSER=466026883 WS=6<br> 27 60.055637 10.1.208.5 10.1.208.39 TCP 43254 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466026883 TSER=354978608<br>
28 60.055699 10.1.208.5 10.1.208.39 TCP 43254 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466026883 TSER=354978608<br> 29 60.056525 10.1.208.39 10.1.208.5 TCP 55672 > 43254 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354978608 TSER=466026883<br>
35 117.235482 10.1.208.39 10.1.208.5 TCP 33070 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=354984326 TSER=466026601<br> 36 117.235554 10.1.208.5 10.1.208.39 TCP 55672 > 33070 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=466032601 TSER=354984326<br>
37 117.236376 10.1.208.39 10.1.208.5 TCP 33070 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=354984326 TSER=466032601<br> 38 117.257662 10.1.208.39 10.1.208.5 TCP 55930 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=354984329 TSER=0 WS=6<br>
39 117.257693 10.1.208.5 10.1.208.39 TCP 55672 > 55930 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=466032603 TSER=354984329 WS=6<br> 40 117.258899 10.1.208.39 10.1.208.5 TCP 55930 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=354984329 TSER=466032603<br>
41 117.258924 10.1.208.39 10.1.208.5 TCP 55930 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=354984329 TSER=466032603<br> 42 117.258931 10.1.208.5 10.1.208.39 TCP 55672 > 55930 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=466032603 TSER=354984329<br>
43 120.056005 10.1.208.5 10.1.208.39 TCP 43254 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=466032883 TSER=354978608<br> 44 120.057752 10.1.208.39 10.1.208.5 TCP 55672 > 43254 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=354984609 TSER=466032883<br>
45 120.057781 10.1.208.5 10.1.208.39 TCP 43254 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=466032883 TSER=354984609<br> 46 120.077268 10.1.208.5 10.1.208.39 TCP 33297 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466032885 TSER=0 WS=6<br>
47 120.079009 10.1.208.39 10.1.208.5 TCP 55672 > 33297 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354984611 TSER=466032885 WS=6<br> 48 120.079030 10.1.208.5 10.1.208.39 TCP 33297 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466032885 TSER=354984611<br>
49 120.079096 10.1.208.5 10.1.208.39 TCP 33297 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466032885 TSER=354984611<br> 50 120.079884 10.1.208.39 10.1.208.5 TCP 55672 > 33297 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354984611 TSER=466032885<br>