[rabbitmq-discuss] flow control issues

[Marek Majkowski]

Romary,

First, thank a lot for the feedback. For example the information about
SSL memory
usage is indeed very interesting. (if that is a big problem to you,
you may always
fall back to the old technique of using stunnel)

The flow control was heavily modified between 1.8.1 and 2.0.0. In summary:
 - 1.8.1 - we have send Channel.flow AMQP message to everyone once
   rabbit reached memory limit
 - 2.0.0 - once we reach memory limit, the connections from which we hear
   publishes are stopped temporarily. We stop receiving bytes from tcp sockets.
   That 'stop' shouldn't take too long, as data should be swapped out to disk
   and memory pressure will drop pretty quickly.


On Wed, Sep 8, 2010 at 11:49, Romary Kremer <romary.kremer at gmail.com> wrote:
> I've started playing a bit with the latest release 2.0.0 and I m affraid
> that it looks like their are some regression or at least some semantic
> updates.

It's best if you upgraded both server and client library. Do you have any
particular problems? A lot was changed in 2.0.0 but we think it's fully
functional. If you found something that blocks you to migrate, you
could report a bug.

> It looks like any listener is called back  when the alarm handler is set or
> cleared, while the producers are still paused / resumed
> like their are to be.

Interesting. Maybe we have a race there? Or maybe you're blocking
the main java client thread? (nothing blocking should be done from
the main thread)

>> during long running tests, we have encountered strange behaviour due to
>> flow control :
>>
>> The queue depth starts to increase linearly for about 2 hours, these is
>> coherent since the message throughput of the single consumer
>> is not enough to absorb message ingress. Memory occupation grow faster as
>> well, until the memory watermark is reached on the broker side.

Are you sure your consumer is ACK-ing the messages it received?

>> From that point, the producers are indeed paused, as flow control request
>> has been issued by the broker, but the consumer seems to be blocked
>> as well. The queue level is flatten at its top value until the end of the
>> test, even when memory occupation lowered under the threshold.

That's how 1.8.1 behaves. In 2.0.0 we introduced swapping out big queues
to disk, so the memory usage shouldn't be dependent on a queue size.

>> By registering the FlowListener callback, we have noticed that not all of
>> the producers are notified all the time the alarm handler is set.
>> Does this mean that the broker applies some heuristic to try not to block
>> every body every time ?
>> Or does it mean that some of the channels have been somehow blacklisted by
>> the broker ?

No, in 1.8.1 broker should send 'channel.flow' to all the channels.

>> Could anybody explain how the blocking of consumer is assumed to be
>> implemented ?

The best description is probably here:
  http://www.rabbitmq.com/extensions.html#memsup

But it covers 2.0.0. I'd suggest an upgrade to 2.0.0 and monitoring
not only queue size but also number of unacknowledged messages
('Msg unack' in status plugin). This number should be near zero.

Cheers,
  Marek Majkowski