[rabbitmq-discuss] of queues, memory and heartbeats...

[Anthony]

Hi all,

We're in a particular situation where occasionally a client will be
disconnected from the server (often through some unclean network break, such
as the random stuff that can happen with 3G data links and virtual
machines), but the queues associated with that client don't die and continue
to grow.

Eventually some of the queues grow so large, that we start seeing txamqp
(the client side library we use) start whining that it doesn't know what to
do with "channel_flow" - presumably this is the memory based throttling
kicking in (though I don't recall explicitly enabling it - unless it's
something auto-enabled by default in the .deb packages?)...

Questions would be:

   1. Even though we haven't explicitly enabled it, if we do start to see
   channel_flow messages, this would be the memory based throttling, correct?

   2. Besides rabbit noticing when a TCP connection dies - is there any
   "heartbeat" functionality built into rabbit and/or the AMQP spec that would
   actually test whether or not a connection is active, and if it isn't, kill
   the relevant queues? Do I as the person who monitors the server configure
   this, or do the programmers who write our clients need to incorporate
   something into their code?

   3. In calculating the "5%" of memory available threshold, what would
   rabbit be considering memory.. Real? Swap? Virtual? All of the above?
   Ie.
   # cat /proc/meminfo | grep -i "total\|free"
   MemTotal:       514056 kB
   MemFree:        144860 kB
   HighTotal:           0 kB
   HighFree:            0 kB
   LowTotal:       514056 kB
   LowFree:        144860 kB
   SwapTotal:     1044184 kB
   SwapFree:       838244 kB
   VmallocTotal:   442360 kB
   HugePages_Total:     0
   HugePages_Free:      0

As a failsafe against runaway queues, I'm thinking of implementing something
that culls queues over a given order of magnitude on a timed basis, or
triggered by a lower memory threshold than rabbit's..
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090901/20bf43e7/attachment.htm