[rabbitmq-discuss] heartbeat bug

Mon Mar 26 21:01:15 BST 2012

I think I've encountered a bug in the implementation of heartbeats,
repeatable in 2.5.1 and 2.7.1.

First, my observation is that assuming a heartbeat interval H seconds,
Rabbit only sends a heartbeat after H seconds of no traffic through a
connection. I've confirmed that this is valid behavior according to
the spec (sec 4.2.7). Upon a closer reading of the spec, I see that
the client "should" be sending heartbeat frames, but it is not
required to do so. At this time, haigha only responds to heartbeats
and does not initiate a client-side loop. As far as I know and have
observed in other implementations, this is standard practice. In other
words, I think both client and server are operating within spec.

So given this, I think there is a timing bug in RabbitMQ where sparse
traffic throws off the timer.  I've pasted example output, showing the
timestamps when we handled a heartbeat (interval=3) and when we
consumed an application message. This is a rare case in that we still
read a heartbeat frame at the very end; usually there are no heartbeat
frames between the final one and any application messages we process
before Rabbit kills the connection.

DEBUG: 1332791022.6 consume
DEBUG: 1332791025.57 heartbeat
DEBUG: 1332791025.72 consume
DEBUG: 1332791028.56 heartbeat
DEBUG: 1332791029.23 consume
DEBUG: 1332791030.16 consume
DEBUG: 1332791030.18 consume
DEBUG: 1332791033.06 heartbeat
DEBUG: 1332791034.06 consume
DEBUG: 1332791034.07 consume
DEBUG: 1332791036.07 heartbeat
DEBUG: 1332791039.07 consume
DEBUG: 1332791039.61 consume
DEBUG: 1332791042.07 heartbeat
DEBUG: 1332791044.74 consume
DEBUG: 1332791045.17 consume
DEBUG: 1332791046.26 consume
DEBUG: 1332791047.67 consume
DEBUG: 1332791049.58 heartbeat
DEBUG: 1332791049.58 transport closed

=ERROR REPORT==== 26-Mar-2012::15:44:09 ===
exception on TCP connection <0.2539.25> from 10.122.166.168:54322
{timeout,running}

-- 
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aaron at agoragames.com
www.agoragames.com