[rabbitmq-discuss] Unable to get Federation connection established
Matt Pietrek
mpietrek at skytap.com
Wed Nov 14 20:01:52 GMT 2012
I've got a head scratcher here with regards to federation.
We have two brokers running 2.8.6. I want to establish bidirectional
federated topic exchanges between the brokers. However, when I start up all
the brokers the federation tab in the Web UI shows "starting" indefinitely.
Looking at the logs, I see a stream of messages like this from each broker:
<<< from host util1 >>>
=WARNING REPORT==== 14-Nov-2012::11:49:19 ===
Federation exchange 'skytap' in vhost '/' did not connect to
west1util1:55672:/:skytap
{error,handshake_receive_timed_out}
<<< from host west1util1 >>>
=WARNING REPORT==== 14-Nov-2012::10:41:41 ===
Federation exchange 'skytap' in vhost '/' did not connect to
util1:55672:/:skytap
{error,handshake_receive_timed_out}
Using Wireshark, I've verified that traffic on port 55672 is going back and
forth between the two machines, corresponding to the 1 minute connection
retry intervals. See very end of this message for packet summary.
Both brokers appear otherwise healthy and the "Exchanges" tab shows "skytap
x-federation D Args" on both brokers.
This exact same setup has worked previously on several different testbeds.
My rabbitmq.config file is generated programmatically and has worked
reliably in other federation scenarios, so I'm reasonably sure it's not
just a fat-fingered rabbitmq.config file. That said, the config file looks
like this:
[
{rabbit, [{cluster_nodes, [rabbit at util1]}, {disk_free_limit, 104857600}]},
{mnesia, [{debug, none}]},
{rabbitmq_federation, [{exchanges, [[{type, "topic"}, {upstream_set,
"skytap_upstreams"}, {virtual_host, "/"}, {exchange, "skytap"}]]},
{upstream_sets, [{connection, "upstream-abc123-cloud/west1"}, {max_hops,
1}, {exchange, "skytap"}]]}]},
{connections, [{"upstream-abc123-cloud/west1", [{username, "foo"}, {host,
"west1util1"}, {password, "foo"}, {port, 55672}]}]}]}
I don't see anything else interesting in the other log files. Any hints on
how I can track this down?
Thanks,
Matt
Packet summary:
No. Time Source Destination Protocol
Info
1 0.000000 10.1.208.5 10.1.208.39 TCP
40729 > 55672 [FIN, ACK] Seq=1 Ack=1 Win=92 Len=0 TSV=466020877
TSER=354966603
2 0.001593 10.1.208.39 10.1.208.5 TCP
55672 > 40729 [FIN, ACK] Seq=1 Ack=2 Win=91 Len=0 TSV=354972603
TSER=466020877
3 0.001607 10.1.208.5 10.1.208.39 TCP
40729 > 55672 [ACK] Seq=2 Ack=2 Win=92 Len=0 TSV=466020878 TSER=354972603
4 0.020673 10.1.208.5 10.1.208.39 TCP
42451 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466020880 TSER=0 WS=6
5 0.022202 10.1.208.39 10.1.208.5 TCP
55672 > 42451 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354972605
TSER=466020880 WS=6
6 0.022215 10.1.208.5 10.1.208.39 TCP
42451 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466020880 TSER=354972605
7 0.022280 10.1.208.5 10.1.208.39 TCP
42451 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466020880
TSER=354972605
8 0.023274 10.1.208.39 10.1.208.5 TCP
55672 > 42451 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354972605 TSER=466020880
14 57.211788 10.1.208.39 10.1.208.5 TCP
56169 > 55672 [FIN, ACK] Seq=1 Ack=1 Win=92 Len=0 TSV=354978324
TSER=466020599
15 57.211854 10.1.208.5 10.1.208.39 TCP
55672 > 56169 [FIN, ACK] Seq=1 Ack=2 Win=91 Len=0 TSV=466026599
TSER=354978324
16 57.212561 10.1.208.39 10.1.208.5 TCP
56169 > 55672 [ACK] Seq=2 Ack=2 Win=92 Len=0 TSV=354978324 TSER=466026599
17 57.231459 10.1.208.39 10.1.208.5 TCP
33070 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=354978326 TSER=0 WS=6
18 57.231491 10.1.208.5 10.1.208.39 TCP
55672 > 33070 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=466026601
TSER=354978326 WS=6
19 57.232238 10.1.208.39 10.1.208.5 TCP
33070 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=354978326 TSER=466026601
20 57.232266 10.1.208.39 10.1.208.5 TCP
33070 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=354978326
TSER=466026601
21 57.232273 10.1.208.5 10.1.208.39 TCP
55672 > 33070 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=466026601 TSER=354978326
22 60.023027 10.1.208.5 10.1.208.39 TCP
42451 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=466026880
TSER=354972605
23 60.024243 10.1.208.39 10.1.208.5 TCP
55672 > 42451 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=354978605
TSER=466026880
24 60.024259 10.1.208.5 10.1.208.39 TCP
42451 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=466026880 TSER=354978605
25 60.054155 10.1.208.5 10.1.208.39 TCP
43254 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466026883 TSER=0 WS=6
26 60.055619 10.1.208.39 10.1.208.5 TCP
55672 > 43254 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354978608
TSER=466026883 WS=6
27 60.055637 10.1.208.5 10.1.208.39 TCP
43254 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466026883 TSER=354978608
28 60.055699 10.1.208.5 10.1.208.39 TCP
43254 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466026883
TSER=354978608
29 60.056525 10.1.208.39 10.1.208.5 TCP
55672 > 43254 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354978608 TSER=466026883
35 117.235482 10.1.208.39 10.1.208.5 TCP
33070 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=354984326
TSER=466026601
36 117.235554 10.1.208.5 10.1.208.39 TCP
55672 > 33070 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=466032601
TSER=354984326
37 117.236376 10.1.208.39 10.1.208.5 TCP
33070 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=354984326 TSER=466032601
38 117.257662 10.1.208.39 10.1.208.5 TCP
55930 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=354984329 TSER=0 WS=6
39 117.257693 10.1.208.5 10.1.208.39 TCP
55672 > 55930 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=466032603
TSER=354984329 WS=6
40 117.258899 10.1.208.39 10.1.208.5 TCP
55930 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=354984329 TSER=466032603
41 117.258924 10.1.208.39 10.1.208.5 TCP
55930 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=354984329
TSER=466032603
42 117.258931 10.1.208.5 10.1.208.39 TCP
55672 > 55930 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=466032603 TSER=354984329
43 120.056005 10.1.208.5 10.1.208.39 TCP
43254 > 55672 [FIN, ACK] Seq=9 Ack=1 Win=5888 Len=0 TSV=466032883
TSER=354978608
44 120.057752 10.1.208.39 10.1.208.5 TCP
55672 > 43254 [FIN, ACK] Seq=1 Ack=10 Win=5824 Len=0 TSV=354984609
TSER=466032883
45 120.057781 10.1.208.5 10.1.208.39 TCP
43254 > 55672 [ACK] Seq=10 Ack=2 Win=5888 Len=0 TSV=466032883 TSER=354984609
46 120.077268 10.1.208.5 10.1.208.39 TCP
33297 > 55672 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=466032885 TSER=0 WS=6
47 120.079009 10.1.208.39 10.1.208.5 TCP
55672 > 33297 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=354984611
TSER=466032885 WS=6
48 120.079030 10.1.208.5 10.1.208.39 TCP
33297 > 55672 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=466032885 TSER=354984611
49 120.079096 10.1.208.5 10.1.208.39 TCP
33297 > 55672 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=8 TSV=466032885
TSER=354984611
50 120.079884 10.1.208.39 10.1.208.5 TCP
55672 > 33297 [ACK] Seq=1 Ack=9 Win=5824 Len=0 TSV=354984611 TSER=466032885
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121114/d63a2bcb/attachment.htm>
More information about the rabbitmq-discuss
mailing list