[rabbitmq-discuss] behavior when client socket close

[Aaron Westendorf]

Opposite direction; the clients where turned off, rabbit continued
serving our infrastructure.

This is an important problem for us because our monitoring tools freak
out when the running clients do not match the number of connections
reported by rabbit.

If the problem is not in our networking setup, then it likely lies
somewhere between linux and rabbit. When I've written epoll network
apps, I've found that I always am notified of a read event when the
socket closes, and then if no data is read, the socket is closed. I'm
wondering if rabbit is making that assumption as well, because if it
is, then the problem lies elsewhere.

-Aaron



On Fri, Sep 16, 2011 at 8:16 AM, Marek Majkowski <majek04 at gmail.com> wrote:
> On Tue, Sep 13, 2011 at 21:17, Aaron Westendorf <aaron at agoragames.com> wrote:
>> We're trying to determine Rabbits expected behavior when a client host
>> shuts off. Server is hardware 2.5.1, client is a VM that was shut down
>> cleanly (i.e. power off). We have no heartbeats, and the client turns
>> on keepalives. We found that linux reported these sockets as
>> ESTABLISHED after the clients were turned off, and we're trying to
>> determine if this is a problem with our vendor's network or expected
>> behavior. Host OS is Linux 2.6.35.
>
> What should happen when you type 'shutdown':
>  1) linux starts shutting down process
>  2) as one of the shutdown steps rabbitmq is cleanly closed
>   (equivalent to /etc/init.d/rabbitmq-server stop)
>  3) during shutdown all network connections get properly closed
>
> The behaviour you're explaining may happen, for example
> when the server is closed uncleanly - 'power cord' shutdown kind
> of thing. TCP/IP is not able to distinguish transient network
> failure from remote host being completely down.
>
> But, it is quite possible that 2) took more time than expected,
> which could be what you're seeing. Rabbitmq sometimes
> needs quite some time to dump messages to the disk
> and shutdown cleanly. In such case operating system won't
> wait too long and will shutdown with rabbitmq and its connections
> still up.
>
> As a workaround:
>  - stop rabbitmq manually before shutting down the system
>  (and make sure it has enough time to write messages)
>  - use keepalives/heartbeats so that clients can detect
>   this kind of issues earlier.
>
> Cheers,
>  Marek
>



-- 
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aaron at agoragames.com
www.agoragames.com