[rabbitmq-discuss] Socket errors and connections not closing

[Chris Duncan]

Hi Mark and Matthias

On 12 Aug 2009, at 05:46, Matthias Radestock wrote:

> Mark,
>
> Mark Westling wrote:
>> I'm using RabbitMQ for background processing in a Ruby on Rails
>> environment.
>> [...]
>> Everything works perfectly on two different deployments of this  
>> system
>> but on the third, the clients are getting occasional errors,  
>> "Connection
>> closed unexpectedly by peer".  At the same time, I'm not seeing any
>> errors at all in the RabbitMQ log on the other server.
>>
>> When the client gets an error sending a message, it creates a new
>> connection and tries passing the message again.  I do see the new
>> connection in rabbit.log but nothing corresponding to the error  
>> reported
>> by the client.  The result is a slow but steadily growing number of
>> sockets in use by RabbitMQ; lsof shows an increase of about  
>> 150-200 of
>> these sockets per day with state "established".  I haven't watched
>> carefully but it appears these sockets aren't dying, even though they
>> refer to ports that aren't open any more on the client machine.
>
> Did you check with lsof/netstat on the *client* machine whether the
> sockets are really dead there?
>
> Also, are these connections going via some kind of firewall or
> loadbalancer? If so, perhaps the connections to the server are kept  
> open
> there.
>
>> If that fails, is there a configuration option that lets
>> RabbitMQ close broken connections?
>
> AMQP's heartbeat mechanism is designed precisely for this  
> situation. The
> heartbeat frequency is negotiated between the client and the server at
> connection establishment. I don't know anything about the Ruby client,
> but I am assuming from the above that it disables heartbeats. If  
> so, try
> to find a way to enable them.
>
> Regards,
>
> Matthias.
>

Mark, I have recently added a heartbeat parameter to the Client#new  
method in Bunny on GitHub (master branch) in preparation for the next  
release.  As I understand it (correct me if I'm wrong Matthias), when  
the client specifies a heartbeat interval in the Connection::TuneOk  
method, the server sends heartbeat frames to the client and the  
client has to either send a heartbeat or other type of frame back to  
the server within 2 or so heartbeat intervals. If the client fails to  
respond in time then the server closes the connection and socket.

I don't know how this would fit in with your retry strategy Mark, but  
the functionality is there if you want to try it out.

Regards,

Chris