[rabbitmq-discuss] Socket errors and connections not closing
celldee at gmail.com
Wed Aug 12 06:54:45 BST 2009
Hi Mark and Matthias
On 12 Aug 2009, at 05:46, Matthias Radestock wrote:
> Mark Westling wrote:
>> I'm using RabbitMQ for background processing in a Ruby on Rails
>> Everything works perfectly on two different deployments of this
>> but on the third, the clients are getting occasional errors,
>> closed unexpectedly by peer". At the same time, I'm not seeing any
>> errors at all in the RabbitMQ log on the other server.
>> When the client gets an error sending a message, it creates a new
>> connection and tries passing the message again. I do see the new
>> connection in rabbit.log but nothing corresponding to the error
>> by the client. The result is a slow but steadily growing number of
>> sockets in use by RabbitMQ; lsof shows an increase of about
>> 150-200 of
>> these sockets per day with state "established". I haven't watched
>> carefully but it appears these sockets aren't dying, even though they
>> refer to ports that aren't open any more on the client machine.
> Did you check with lsof/netstat on the *client* machine whether the
> sockets are really dead there?
> Also, are these connections going via some kind of firewall or
> loadbalancer? If so, perhaps the connections to the server are kept
>> If that fails, is there a configuration option that lets
>> RabbitMQ close broken connections?
> AMQP's heartbeat mechanism is designed precisely for this
> situation. The
> heartbeat frequency is negotiated between the client and the server at
> connection establishment. I don't know anything about the Ruby client,
> but I am assuming from the above that it disables heartbeats. If
> so, try
> to find a way to enable them.
Mark, I have recently added a heartbeat parameter to the Client#new
method in Bunny on GitHub (master branch) in preparation for the next
release. As I understand it (correct me if I'm wrong Matthias), when
the client specifies a heartbeat interval in the Connection::TuneOk
method, the server sends heartbeat frames to the client and the
client has to either send a heartbeat or other type of frame back to
the server within 2 or so heartbeat intervals. If the client fails to
respond in time then the server closes the connection and socket.
I don't know how this would fit in with your retry strategy Mark, but
the functionality is there if you want to try it out.
More information about the rabbitmq-discuss