[rabbitmq-discuss] Socket errors and connections not closing

[Mark Westling]

Hi all,
I'm using RabbitMQ for background processing in a Ruby on Rails environment.
 The client application runs in multiple instances via Passenger/nginx on a
CentOS 5 server and uses the Bunny gem to send messages to RabbitMQ running
on a different CentOS server.  Each of the 15 instances of the RoR
application on the client machine opens a connection and keeps it in a
threadsafe location for repeated use.

Everything works perfectly on two different deployments of this system but
on the third, the clients are getting occasional errors, "Connection closed
unexpectedly by peer".  At the same time, I'm not seeing any errors at all
in the RabbitMQ log on the other server.

When the client gets an error sending a message, it creates a new connection
and tries passing the message again.  I do see the new connection in
rabbit.log but nothing corresponding to the error reported by the client.
 The result is a slow but steadily growing number of sockets in use by
RabbitMQ; lsof shows an increase of about 150-200 of these sockets per day
with state "established".  I haven't watched carefully but it appears these
sockets aren't dying, even though they refer to ports that aren't open any
more on the client machine.

Since this is happening in only one deployment, my first action was to look
for differences in configuration between a working client/server pair and a
problem pair, but I'm at a loss to find any.  I know I must be overlooking
something.  I've compared ruby gem versions, RabbitMQ versions, Erlang
versions, kernel versions, kernel network configuration variables, etc.
 I've used netstat -s to look for unusual network problems but I don't see
any.

Any suggestions on where to look for the cause of these broken connections?
 If that fails, is there a configuration option that lets RabbitMQ close
broken connections?

Thanks,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090812/cd716542/attachment.htm