[rabbitmq-discuss] Socket errors and connections not closing

[Matthias Radestock]

Mark,

Mark Westling wrote:
> I'm using RabbitMQ for background processing in a Ruby on Rails 
> environment.
> [...]
> Everything works perfectly on two different deployments of this system 
> but on the third, the clients are getting occasional errors, "Connection 
> closed unexpectedly by peer".  At the same time, I'm not seeing any 
> errors at all in the RabbitMQ log on the other server.  
> 
> When the client gets an error sending a message, it creates a new 
> connection and tries passing the message again.  I do see the new 
> connection in rabbit.log but nothing corresponding to the error reported 
> by the client.  The result is a slow but steadily growing number of 
> sockets in use by RabbitMQ; lsof shows an increase of about 150-200 of 
> these sockets per day with state "established".  I haven't watched 
> carefully but it appears these sockets aren't dying, even though they 
> refer to ports that aren't open any more on the client machine.

Did you check with lsof/netstat on the *client* machine whether the 
sockets are really dead there?

Also, are these connections going via some kind of firewall or 
loadbalancer? If so, perhaps the connections to the server are kept open 
there.

> If that fails, is there a configuration option that lets 
> RabbitMQ close broken connections?

AMQP's heartbeat mechanism is designed precisely for this situation. The 
heartbeat frequency is negotiated between the client and the server at 
connection establishment. I don't know anything about the Ruby client, 
but I am assuming from the above that it disables heartbeats. If so, try 
to find a way to enable them.

Regards,

Matthias.