[rabbitmq-discuss] SSL connection errors once or twice every week

Brian Jing brian.newsgroup at gmail.com
Mon Mar 5 05:33:43 GMT 2012


Hello,

We have been using RabbitMQ together with federation plugin in our
production environment. Every few days we have some downstream servers
and/or upstream servers hanging, which cannot recover on their own and have
to be killed and restarted manually. Normal restart (service
rabbitmq-server restart) on these hanging servers would just stay there
forever trying to restart, so they have to be force killed before being
restarted.

We have got multiple upstream and downstream servers. Each downstream
server talks to all upstream servers, and vice versa. We use federation in
a "symmetric" way in that federation exchanges on the servers are used for
both downstream and upstream purposes. max_hops is set to 1 so it doesn't
form a loop. However, there has been a lot of traffic generated by RabbitMQ
on the network. I've read a recent discussion on symmetric federation on
this mailing list and it seems that there are some problems with it. Not
sure if the error we've been having has anything to do with it.

It seems to me that the error message suggests a connection problem, but
not sure whether it has to do with the federation plugin. Since the error
will take a while to happen again (but will definitely happen) and it'll
probably take at least a couple of weeks for us to determine the cause of
it, I was just wondering if anyone can easily see what this error is about
and if there is any fix for this.

Let me know if more information is needed. Any help would be greatly
appreciated!

Here is the error message:

=INFO REPORT==== 17-Feb-2012::11:58:39 ===
starting TCP connection <0.19666.4> from 10.19.100.10:51445

=INFO REPORT==== 17-Feb-2012::11:58:40 ===
upgraded TCP connection <0.19666.4> to SSL

=ERROR REPORT==== 17-Feb-2012::11:58:41 ===
connection <0.19666.4>, channel 2 - error:
{amqp_error,not_found,
            "no exchange 'federation: ts -> chewbacca:rs B' in vhost '/'",
            'exchange.delete'}

=INFO REPORT==== 17-Feb-2012::11:59:55 ===
Federation exchange 'ts' in vhost '/' connected to 10.19.100.10:5671:/:rs

=INFO REPORT==== 19-Feb-2012::00:17:56 ===
SSL: Socket error: ehostunreach


=ERROR REPORT==== 19-Feb-2012::00:17:56 ===
** State machine <0.19667.4> terminating
** Last message in was {tcp_error,#Port<0.7174>,ehostunreach}
** When State == connection
**      Data  == {state,server,
                     {#Ref<0.0.2.41472>,<0.19666.4>},
                     gen_tcp,tcp,tcp_closed,tcp_error,"localhost",5671,
                     #Port<0.7174>,
                     {ssl_options,[],verify_peer,
                         {#Fun<ssl.3.66992289>,
                          #Fun<rabbit_networking.0.80337212>},
                         false,false,undefined,1,
                         "/etc/rabbitmq/certs/server/cert.pem",undefined,
                         "/etc/rabbitmq/certs/server/key.pem",undefined,
                         undefined,undefined,
                         "/etc/rabbitmq/certs/testca/cacert.pem",undefined,
                         undefined,
                         [<<0,57>>,
                          <<0,56>>,
                          <<0,53>>,
                          <<0,22>>,
                          <<0,19>>,
                          <<0,10>>,
                          <<0,51>>,
                          <<0,50>>,
                          <<0,47>>,
                          <<0,5>>,
                          <<0,4>>,
                          <<0,21>>,
                          <<0,9>>],
                         #Fun<ssl.0.123982462>,true,18446744073709551900,
                         false,[],undefined},
                     {socket_options,binary,0,0,0,once},
                     {connection_states,
                         {connection_state,
                             {security_parameters,
                                 <<0,53>>,
                                 0,7,1,16,256,32,unknown,2,20,0,

<<213,226,167,243,36,225,117,174,26,113,107,

68,197,183,4,70,236,126,250,31,203,52,69,43,

105,252,205,75,169,13,117,64,134,62,230,13,

229,241,170,181,103,116,249,98,34,110,81,120>>,

<<79,61,166,54,4,241,68,223,169,240,194,203,

212,204,88,120,161,154,112,195,214,164,224,
                                   199,236,47,165,172,151,235,119,66>>,

<<79,61,166,63,26,160,125,193,150,16,70,58,16,

153,134,49,222,19,30,138,216,70,224,16,212,
                                   31,66,80,249,190,233,232>>,
                                 undefined},
                             undefined,
                             {cipher_state,

<<166,237,209,135,65,44,54,254,116,232,232,21,
                                   71,65,156,243>>,

<<222,58,54,75,192,174,82,129,229,82,125,31,

199,227,93,36,190,91,163,225,12,12,107,218,
                                   46,37,147,165,242,251,96,61>>,
                                 undefined},

<<155,100,115,240,203,42,54,112,212,4,250,136,248,
                               33,61,146,46,52,250,92>>,
                             4726,true,
                             <<68,159,4,227,232,222,225,139,24,69,55,147>>,

<<53,139,69,42,122,101,224,100,144,98,247,136>>},
                         {connection_state,
                             {security_parameters,undefined,0,undefined,
                                 undefined,undefined,undefined,undefined,
                                 undefined,undefined,undefined,undefined,
                                 undefined,undefined,

<<79,61,166,64,135,104,6,75,30,243,86,193,245,

128,225,5,92,4,97,194,111,172,21,242,129,27,
                                   88,20,111,83,220,189>>,
                                 undefined},
                             undefined,undefined,undefined,undefined,true,
                             undefined,undefined},
                         {connection_state,
                             {security_parameters,
                                 <<0,53>>,
                                 0,7,1,16,256,32,unknown,2,20,0,

<<213,226,167,243,36,225,117,174,26,113,107,

68,197,183,4,70,236,126,250,31,203,52,69,43,

105,252,205,75,169,13,117,64,134,62,230,13,

229,241,170,181,103,116,249,98,34,110,81,120>>,

<<79,61,166,54,4,241,68,223,169,240,194,203,

212,204,88,120,161,154,112,195,214,164,224,
                                   199,236,47,165,172,151,235,119,66>>,

<<79,61,166,63,26,160,125,193,150,16,70,58,16,

153,134,49,222,19,30,138,216,70,224,16,212,
                                   31,66,80,249,190,233,232>>,
                                 undefined},
                             undefined,
                             {cipher_state,
                                 <<22,11,209,109,83,84,156,48,32,78,217,242,
                                   122,245,245,181>>,

<<6,203,146,213,209,163,106,164,13,97,19,252,
                                   140,9,96,227,103,97,128,19,70,78,185,136,
                                   113,204,139,66,127,138,64,34>>,
                                 undefined},

<<5,194,148,88,152,255,80,36,119,74,250,112,76,87,
                               214,237,242,76,135,6>>,
                             7060,true,
                             <<68,159,4,227,232,222,225,139,24,69,55,147>>,

<<53,139,69,42,122,101,224,100,144,98,247,136>>},
                         {connection_state,
                             {security_parameters,undefined,0,undefined,
                                 undefined,undefined,undefined,undefined,
                                 undefined,undefined,undefined,undefined,
                                 undefined,undefined,

<<79,61,166,64,96,136,174,122,110,176,1,184,

198,114,174,129,32,102,231,33,11,242,30,88,
                                   113,226,126,206,155,112,239,90>>,
                                 undefined},
                             undefined,undefined,undefined,undefined,true,
                             undefined,undefined}},
                     [],<<>>,<<>>,
                     {<<>>,<<>>},
                     [],2994260,
                     {session,

<<138,157,166,139,130,58,93,77,177,4,153,202,214,175,

148,97,39,47,47,26,51,187,222,253,146,247,213,191,
                           204,176,112,194>>,

<<48,130,2,225,48,130,1,201,160,3,2,1,2,2,1,2,48,13,6,

9,42,134,72,134,247,13,1,1,5,5,0,48,19,49,17,48,15,

6,3,85,4,3,19,8,77,121,84,101,115,116,67,65,48,30,

23,13,49,49,49,50,48,49,48,54,49,48,51,54,90,23,13,

50,49,49,49,50,56,48,54,49,48,51,54,90,48,36,49,17,

48,15,6,3,85,4,3,19,8,103,114,111,112,101,108,108,

111,49,15,48,13,6,3,85,4,10,19,6,99,108,105,101,110,

116,48,130,1,34,48,13,6,9,42,134,72,134,247,13,1,1,

1,5,0,3,130,1,15,0,48,130,1,10,2,130,1,1,0,187,221,

90,147,108,212,164,227,233,68,216,216,245,254,200,2,
                           41,241,207,145,28,253,107,144,64,25,91,82,249,63,

162,128,86,104,9,161,128,48,63,213,237,57,96,170,74,

150,160,235,155,128,136,187,91,17,243,88,199,163,39,

146,182,6,75,3,206,25,59,91,121,71,161,27,44,132,99,

226,218,92,244,179,186,225,169,216,212,111,193,106,

96,102,224,158,142,64,25,57,42,111,162,148,203,189,

122,113,75,191,250,49,224,230,76,32,19,208,73,252,
                           37,216,63,79,227,15,205,53,156,63,248,27,174,193,

194,17,25,103,228,227,136,37,67,51,140,149,220,54,
                           176,223,163,138,228,181,18,48,41,199,206,179,155,

153,202,22,38,220,212,136,198,25,21,85,182,135,121,

140,45,118,254,171,132,43,24,244,55,243,19,171,180,

48,4,217,195,178,225,224,35,17,13,123,251,90,89,13,

185,10,166,52,58,219,181,27,43,13,107,136,125,101,

252,4,144,17,236,91,75,1,211,167,171,234,32,144,18,

13,236,231,64,112,196,185,114,155,223,217,64,95,103,

100,82,177,156,51,192,78,190,185,247,90,142,111,2,3,

1,0,1,163,47,48,45,48,9,6,3,85,29,19,4,2,48,0,48,11,

6,3,85,29,15,4,4,3,2,7,128,48,19,6,3,85,29,37,4,12,

48,10,6,8,43,6,1,5,5,7,3,2,48,13,6,9,42,134,72,134,

247,13,1,1,5,5,0,3,130,1,1,0,70,235,45,39,71,230,62,

77,167,49,17,222,181,4,192,226,130,231,24,217,253,

138,40,88,134,118,159,17,197,141,12,244,197,11,57,

71,161,42,190,181,77,162,212,204,46,207,197,224,14,

182,125,37,50,28,201,156,79,128,190,195,124,96,107,

243,16,2,224,88,247,147,219,13,58,107,159,143,234,

243,201,107,56,167,19,22,166,184,43,150,77,97,190,

183,210,96,230,39,64,33,77,37,249,200,244,145,196,

77,58,215,201,167,210,180,82,74,231,129,26,165,82,

67,230,198,253,185,224,98,234,75,139,169,209,223,80,

93,179,198,235,16,189,177,172,43,80,85,151,39,143,

85,84,57,217,167,99,140,170,59,191,153,149,3,66,15,

126,63,112,142,182,0,170,84,15,13,251,189,171,132,

149,185,67,172,53,101,244,100,143,47,56,159,112,250,

36,22,183,21,205,192,183,243,87,0,13,180,42,110,64,

108,224,208,160,47,32,96,204,2,99,34,124,50,23,159,
                           61,173,153,40,121,13,53,161,0,153,83,210,128,118,

210,43,47,186,115,165,218,0,60,125,238,54,166,161,
                           59,132,139,111,212,1,162,16>>,

<<48,130,2,225,48,130,1,201,160,3,2,1,2,2,1,1,48,13,6,

9,42,134,72,134,247,13,1,1,5,5,0,48,19,49,17,48,15,

6,3,85,4,3,19,8,77,121,84,101,115,116,67,65,48,30,

23,13,49,49,49,50,48,49,48,54,48,56,52,55,90,23,13,

50,49,49,49,50,56,48,54,48,56,52,55,90,48,36,49,17,

48,15,6,3,85,4,3,19,8,103,114,111,112,101,108,108,
                           111,49,15,48,13,6,3,85,4,10,19,6,115,101,114,118,

101,114,48,130,1,34,48,13,6,9,42,134,72,134,247,13,

1,1,1,5,0,3,130,1,15,0,48,130,1,10,2,130,1,1,0,211,

197,239,159,234,44,225,233,71,198,112,181,20,65,254,

38,226,54,237,75,111,195,171,229,251,172,239,71,237,

207,182,77,39,117,121,116,54,178,177,70,22,78,187,

91,190,144,174,90,147,253,217,24,153,88,86,240,12,

43,192,0,173,226,154,1,40,156,143,116,62,195,6,212,

21,183,62,164,103,209,144,165,14,228,36,214,90,71,

201,119,70,80,98,77,202,255,89,98,130,133,51,2,201,

84,175,115,29,17,87,173,127,119,120,163,168,213,218,

239,59,165,125,226,189,233,156,190,16,69,122,38,36,

45,170,150,164,190,51,14,57,123,156,61,47,125,189,

45,221,226,242,11,36,205,81,219,144,53,42,16,39,39,
                           126,84,159,148,103,193,40,135,227,33,143,125,164,

143,177,194,192,26,171,244,97,230,210,98,111,120,56,

105,67,197,126,115,247,227,91,42,229,14,103,107,249,

108,221,169,209,60,187,142,164,149,95,94,4,168,233,

236,111,140,76,53,127,189,41,245,59,232,208,252,139,

62,63,176,222,57,111,75,23,91,65,121,132,178,235,30,

130,115,169,0,57,82,139,177,132,110,2,208,14,37,2,3,

1,0,1,163,47,48,45,48,9,6,3,85,29,19,4,2,48,0,48,11,

6,3,85,29,15,4,4,3,2,5,32,48,19,6,3,85,29,37,4,12,

48,10,6,8,43,6,1,5,5,7,3,1,48,13,6,9,42,134,72,134,

247,13,1,1,5,5,0,3,130,1,1,0,107,62,132,105,170,9,
                           212,46,223,127,176,167,81,78,149,136,133,186,145,

118,183,245,45,117,104,68,174,83,238,180,145,125,10,

134,100,175,62,192,13,151,174,243,154,118,233,204,

240,45,57,146,157,38,210,154,147,18,248,248,76,105,

71,56,156,145,129,96,236,241,156,93,60,198,120,109,

199,15,2,66,93,229,126,51,228,22,152,253,1,245,58,

23,107,222,158,252,17,174,96,214,174,62,171,184,72,
                           16,246,225,214,2,246,220,11,211,209,14,187,46,74,
                           229,235,242,73,39,218,19,214,198,161,119,141,207,

163,126,109,186,193,57,102,167,206,153,100,146,123,

126,172,120,219,60,31,129,222,176,178,138,162,192,

234,5,124,136,68,141,100,66,74,21,252,121,102,241,

25,95,236,100,4,14,226,92,169,34,115,218,185,195,47,

79,30,156,81,160,132,119,51,22,82,66,109,55,193,43,
                           49,162,245,175,70,229,50,80,178,53,186,14,147,41,

214,80,136,212,85,142,180,149,3,77,240,150,207,150,

32,50,0,170,79,141,97,41,2,150,7,110,28,228,138,89,
                           252,86,203,221,77,55,185,38,127,57,155,132>>,
                         0,
                         <<0,53>>,
                         <<213,226,167,243,36,225,117,174,26,113,107,68,197,

183,4,70,236,126,250,31,203,52,69,43,105,252,205,75,

169,13,117,64,134,62,230,13,229,241,170,181,103,116,
                           249,98,34,110,81,120>>,
                         true,undefined},
                     3006545,ssl_session_cache,
                     {3,1},
                     undefined,false,rsa,undefined,
                     {'RSAPrivateKey','two-prime',

26733888243954032639493945355378651641939366228481256857536474393698815990033421919494691489863707532678433266055379180921670441122471606114607517646228859067337981366652832175793337203970265712841766561330994102352268924716159787538639704745920435882299144174112558557010046099015297165656332137749120327001691289263898507721876568514798182129906643473214153540916185542584190269258525271080383264965658901863970305368253883803888475734975817922342926472990395164199248748259739148561733932569200222557037932550632037883875163976459062154731623262601492909960284943535117190965093684222976486329405290326506754543141,
                         65537,

19638926341103848931073996447645601792110240744983774511333079378217431329236447577584454996231111211274096940063936025242121270692895204308736505020981158596486256235655186417922615108228092107955109469898215970576126998665407278503139752281727779195224224439911424373540010366525664384139622723649139402523057646332031746318146069882155546775041878344935580784164293428918078878647526946291756476712430695689632084526183710423269560979597244472951214040124632189711837793336370653813366701595674106850012255679776174650307416978789128301595245252412894223959611800760267511646456073285111733061886471761081646015681,

177468221471267592959462072764121466940196903068066410572043602454082240715081242042509786132231542521680132746426975245619690593583885962543716788317814592781430786138246575995345414171447935333333738375794904031190813035393698344746110335546405731252630779784661964873323575380259166711051864760760459167039,

150640424647982932748549989167076843675993323800890265692777316272428245602205796254497685083373967363821039945013560477307144689417342997024955062130871951088291677187010112883632338552378119297706080173393731599917709479223740731270666048786064065637674964132109182747349377809019088277441708637239984414619,

11338014301847771666711440845070366084480590096372950563271839777152483968964785700169194112267169210343389471737945669673766643504214879002251280844205406106105721982404419086813727346931542567414870417923512873317300672584851533781710544805725022456852847627422366707731599098480936433147293250428064185611,

99417117763622958134188078969962122811145814441248543139661633633139546802365154029014661568672518370080834028132604754024270596864045032688761407255206579923871883389422179874618397349092244317933127541077951068246040715861043900528233638129527487784926340061980047487501034362202932788743897065442479910601,

57680421567639822270628306189496861089385670094641916193476111860126145371827374999675381364998416489580079100178978901092831951245342360541564455113199735957614397864732338137863696612804836795154913219598727549346153287983388725107880942227515400079146331721681590144968544898949263973602287315085851221482,
                         asn1_NOVALUE},
                     {'DHParameter',

179769313486231590770839156793787453197860296048756011706444423684197180216158519368947833795864925541502180565485980503646440548199239100050792877003355816639229553136239076508735759914822574862575007425302077447712589550957937778424442426617334727629299387668709205606050270810842907692932019128194467627007,
                         2,asn1_NOVALUE},
                     undefined,undefined,#Ref<0.0.0.3248>,undefined,0,<<>>,
                     true,undefined,false,
                     {[],[]},
                     false}
** Reason for termination =
** {badarg,[{ssl_connection,send_user,2},
            {ssl_connection,handle_info,3},
            {gen_fsm,handle_msg,7},
            {proc_lib,init_p_do_apply,3}]}

=ERROR REPORT==== 19-Feb-2012::00:17:56 ===
exception on TCP connection <0.19666.4> from 10.19.100.10:51445
{writer,send_failed,closed}

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120305/65c307ff/attachment.htm>


More information about the rabbitmq-discuss mailing list