[rabbitmq-discuss] behavior when client socket close

[Marek Majkowski]

On Tue, Sep 13, 2011 at 21:17, Aaron Westendorf <aaron at agoragames.com> wrote:
> We're trying to determine Rabbits expected behavior when a client host
> shuts off. Server is hardware 2.5.1, client is a VM that was shut down
> cleanly (i.e. power off). We have no heartbeats, and the client turns
> on keepalives. We found that linux reported these sockets as
> ESTABLISHED after the clients were turned off, and we're trying to
> determine if this is a problem with our vendor's network or expected
> behavior. Host OS is Linux 2.6.35.

What should happen when you type 'shutdown':
 1) linux starts shutting down process
 2) as one of the shutdown steps rabbitmq is cleanly closed
   (equivalent to /etc/init.d/rabbitmq-server stop)
 3) during shutdown all network connections get properly closed

The behaviour you're explaining may happen, for example
when the server is closed uncleanly - 'power cord' shutdown kind
of thing. TCP/IP is not able to distinguish transient network
failure from remote host being completely down.

But, it is quite possible that 2) took more time than expected,
which could be what you're seeing. Rabbitmq sometimes
needs quite some time to dump messages to the disk
and shutdown cleanly. In such case operating system won't
wait too long and will shutdown with rabbitmq and its connections
still up.

As a workaround:
 - stop rabbitmq manually before shutting down the system
  (and make sure it has enough time to write messages)
 - use keepalives/heartbeats so that clients can detect
   this kind of issues earlier.

Cheers,
  Marek