[rabbitmq-discuss] chan.flow and vm_memory_high_watermark

[Simon MacMullen]

On 09/06/10 22:27, Nicolás César wrote:
> (rabbit at sendsorium)1> memory().
> [{total,27420688},
<snip>
> but in top I see beam.smp using:
> VIRT=141m  RES=95m

So there is some disparity here but not as much as when you were seeing 
400M used. I take it this was a different run?

> I've been using:
>   Erlang R13B02 (erts-5.7.3) [source] [smp:2:2] [rq:2] [async-threads:0]
> [kernel-poll:false]
>
> Is that old? I'm trying other resources before changing erlang version
> (and taking my Debian to 'testing' instead of 'stable') . Thanks for the
> advice. Now I'm aware of that... if everything fails, I'll do it!

It's old enough to make a difference. If you're in general concerned 
about memory use then R13B04 should improve things. But see below for 
something more important...

>     * Are you stopping the queues from hibernating? If you've written
>     something like a shell script to repeatedly invoke rabbitmqctl
>     list_queues this will keep the queue processes from hibernating
>     which in turn stops them from GCing on hibernate. If queues are busy
>     this is not a problem as they also GC every (n) reductions, but if
>     they're almost-but-not-quite idle it can be a problem. It doesn't
>     have to be a shell script, anything that means the queues never get
>     10 seconds or so idle.
>
>
> This is really nice data!! Yes. I have scripts but they're cron'ed every
> minute.... I'm changing that and testing again.

Every minute *should* be OK, it's more like every couple of seconds 
which can prevent the hibernation.

>
> One small detail. I'm using:
>
> -----------------------------------------------------
> while True:
>    msg = self.chan.basic_get(self.config['queue'])
>    if msg:
>       ....
>    time.sleep(2.0)
> ----------------------------------------------------
>
> because I need a fixed-rate consumer . ... could this be a similar case
> of the "almost-but-not-quite idle queue"?

Yes, as you'll still be waking the queue up every 2 seconds even if it's 
empty. I suspect this is your real problem.

> is there any way to tell the broker to consume at a fixed rate with
> chan.basic_consume(...callback = f)?

Yes, basic.qos. This not-very-well-named method lets you specify how 
many unacked messages the server will send to you at any time. So set 
qos to a low value, then use basic_consume with a sleep before acking.

This is an interesting case though, I'll file a bug on this behaviour.

Cheers, Simon