[rabbitmq-discuss] Unable to get Federation connection established

Matt Pietrek mpietrek at skytap.com
Wed Nov 14 20:01:52 GMT 2012


I've got a head scratcher here with regards to federation.

We have two brokers running 2.8.6. I want to establish bidirectional
federated topic exchanges between the brokers. However, when I start up all
the brokers the federation tab in the Web UI shows "starting" indefinitely.

Looking at the logs, I see a stream of messages like this from each broker:

<<< from host util1 >>>
=WARNING REPORT==== 14-Nov-2012::11:49:19 ===
Federation exchange 'skytap' in vhost '/' did not connect to
west1util1:55672:/:skytap
{error,handshake_receive_timed_out}


<<< from host west1util1 >>>
=WARNING REPORT==== 14-Nov-2012::10:41:41 ===
Federation exchange 'skytap' in vhost '/' did not connect to
util1:55672:/:skytap
{error,handshake_receive_timed_out}


Using Wireshark, I've verified that traffic on port 55672 is going back and
forth between the two machines, corresponding to the 1 minute connection
retry intervals. See very end of this message for packet summary.

Both brokers appear otherwise healthy and the "Exchanges" tab shows "skytap
x-federation D Args" on both brokers.

This exact same setup has worked previously on several different testbeds.
My rabbitmq.config file is generated programmatically and has worked
reliably in other federation scenarios, so I'm reasonably sure it's not
just a fat-fingered rabbitmq.config file. That said, the config file looks
like this:

[
{rabbit, [{cluster_nodes, [rabbit at util1]}, {disk_free_limit, 104857600}]},
{mnesia, [{debug, none}]},
{rabbitmq_federation, [{exchanges, [[{type, "topic"}, {upstream_set,
"skytap_upstreams"}, {virtual_host, "/"}, {exchange, "skytap"}]]},
{upstream_sets, [{connection, "upstream-abc123-cloud/west1"}, {max_hops,
1}, {exchange, "skytap"}]]}]},
{connections, [{"upstream-abc123-cloud/west1", [{username, "foo"}, {host,
"west1util1"}, {password, "foo"}, {port, 55672}]}]}]}

I don't see anything else interesting in the other log files. Any hints on
how I can track this down?

Thanks,

Matt

Packet summary:

No.     Time        Source                Destination           Protocol
Info
      1 0.000000    10.1.208.5            10.1.208.39           TCP
40729 > 55672 [FIN, ACK] Seq=1 Ack=1 Win=92 Len=0 TSV=466020877
TSER=354966603
      2 0.001593    10.1.208.39           10.1.208.5            TCP
55672 > 40729 [FIN, ACK] Seq=1 Ack=2 Win=91 Len=0 TSV=354972603
TSER=466020877
      3 0.001607    10.1.208.5            10.1.208.39           TCP
40729 > 55672 [ACK] Seq=2 Ack=2 Win=92 Len=0 TSV=466020878 TSER=354972603
      4 0.020673    10.1.208.5            10.1.208.39           TCP
42451 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466020880 TSER=0 WS=6
      5 0.022202    10.1.208.39           10.1.208.5            TCP
55672 > 42451 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354972605
TSER=466020880 WS=6
      6 0.022215    10.1.208.5            10.1.208.39           TCP
42451 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466020880 TSER=354972605
      7 0.022280    10.1.208.5            10.1.208.39           TCP
42451 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466020880
TSER=354972605
      8 0.023274    10.1.208.39           10.1.208.5            TCP
55672 > 42451 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354972605 TSER=466020880
     14 57.211788   10.1.208.39           10.1.208.5            TCP
56169 > 55672 [FIN, ACK] Seq=1 Ack=1 Win=92 Len=0 TSV=354978324
TSER=466020599
     15 57.211854   10.1.208.5            10.1.208.39           TCP
55672 > 56169 [FIN, ACK] Seq=1 Ack=2 Win=91 Len=0 TSV=466026599
TSER=354978324
     16 57.212561   10.1.208.39           10.1.208.5            TCP
56169 > 55672 [ACK] Seq=2 Ack=2 Win=92 Len=0 TSV=354978324 TSER=466026599
     17 57.231459   10.1.208.39           10.1.208.5            TCP
33070 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=354978326 TSER=0 WS=6
     18 57.231491   10.1.208.5            10.1.208.39           TCP
55672 > 33070 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=466026601
TSER=354978326 WS=6
     19 57.232238   10.1.208.39           10.1.208.5            TCP
33070 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=354978326 TSER=466026601
     20 57.232266   10.1.208.39           10.1.208.5            TCP
33070 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=354978326
TSER=466026601
     21 57.232273   10.1.208.5            10.1.208.39           TCP
55672 > 33070 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=466026601 TSER=354978326
     22 60.023027   10.1.208.5            10.1.208.39           TCP
42451 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=466026880
TSER=354972605
     23 60.024243   10.1.208.39           10.1.208.5            TCP
55672 > 42451 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=354978605
TSER=466026880
     24 60.024259   10.1.208.5            10.1.208.39           TCP
42451 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=466026880 TSER=354978605
     25 60.054155   10.1.208.5            10.1.208.39           TCP
43254 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466026883 TSER=0 WS=6
     26 60.055619   10.1.208.39           10.1.208.5            TCP
55672 > 43254 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354978608
TSER=466026883 WS=6
     27 60.055637   10.1.208.5            10.1.208.39           TCP
43254 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466026883 TSER=354978608
     28 60.055699   10.1.208.5            10.1.208.39           TCP
43254 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466026883
TSER=354978608
     29 60.056525   10.1.208.39           10.1.208.5            TCP
55672 > 43254 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354978608 TSER=466026883
     35 117.235482  10.1.208.39           10.1.208.5            TCP
33070 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=354984326
TSER=466026601
     36 117.235554  10.1.208.5            10.1.208.39           TCP
55672 > 33070 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=466032601
TSER=354984326
     37 117.236376  10.1.208.39           10.1.208.5            TCP
33070 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=354984326 TSER=466032601
     38 117.257662  10.1.208.39           10.1.208.5            TCP
55930 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=354984329 TSER=0 WS=6
     39 117.257693  10.1.208.5            10.1.208.39           TCP
55672 > 55930 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=466032603
TSER=354984329 WS=6
     40 117.258899  10.1.208.39           10.1.208.5            TCP
55930 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=354984329 TSER=466032603
     41 117.258924  10.1.208.39           10.1.208.5            TCP
55930 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=354984329
TSER=466032603
     42 117.258931  10.1.208.5            10.1.208.39           TCP
55672 > 55930 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=466032603 TSER=354984329
     43 120.056005  10.1.208.5            10.1.208.39           TCP
43254 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=466032883
TSER=354978608
     44 120.057752  10.1.208.39           10.1.208.5            TCP
55672 > 43254 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=354984609
TSER=466032883
     45 120.057781  10.1.208.5            10.1.208.39           TCP
43254 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=466032883 TSER=354984609
     46 120.077268  10.1.208.5            10.1.208.39           TCP
33297 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466032885 TSER=0 WS=6
     47 120.079009  10.1.208.39           10.1.208.5            TCP
55672 > 33297 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354984611
TSER=466032885 WS=6
     48 120.079030  10.1.208.5            10.1.208.39           TCP
33297 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466032885 TSER=354984611
     49 120.079096  10.1.208.5            10.1.208.39           TCP
33297 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466032885
TSER=354984611
     50 120.079884  10.1.208.39           10.1.208.5            TCP
55672 > 33297 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354984611 TSER=466032885
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121114/d63a2bcb/attachment.htm>


More information about the rabbitmq-discuss mailing list