[rabbitmq-discuss] flow control issues

[Marek Majkowski]

On Wed, Sep 15, 2010 at 19:11, romary.kremer at gmail.com
<romary.kremer at gmail.com> wrote:
> Yes. You will never hear "FlowListener.handleFlow()" and it may be possible
> for
> channel.publish to block (though I would need to consult the sources
> to be sure).
>
> It seems to me that FlowListener interface is likely to be deprecated so,
> does'nt it ?

It's more complex than that, but yes, you won't hear anything on that interface.

> Does not really matter for us anyway, cause we where on wrong idea using
> that.
> Does this new implementation keep the broker on track for compliance with
> specification then ?

Yes, the spec doesn't force us to send channel.flow. We implemented that
for a while but realized that it doesn't solve our problems.

> This would be acceptable for our needs, only if we can somehow guarantee
> that's an upper boundary !

Optimistically, the upper bouadary is not more than your memory usage
divided by the disk throughput.

> But it is possible to create a very pessimistic environment in which the
> memory usage will not drop - and the connection could be stuck for a long
> time.
> (though it's pretty unlikely).

> ... Not that much unlikely, considering my little playing with the
> MultiCastMain sample (see my previous reply about it for details).
> I get 100 % times blocked connection.
> What would be, based on your knowledge and your intuition, "a very
> pessimistic environment in which the memory usage will not drop" ?
> I think that the experimentation I've done on the MultiCastMain is maybe a
> beginning of an answer for that question, although I would
> never have considered that a single producer could have such power to flood
> the broker.

Okay. The memory can stay high due to a lot of reasons. If your metadata
that rabbit never releases is using more memory than threshold -
rabbit will just get stuck.

Next thing. remember, we don't control how erlang eats memory - and it
has pretty
complex GC and memory allocation mechanisms.

If you think that you have enough memory for all the connections,
queues, exchanges and bindings. And some memory for the messages.
And you still hit the limit when you get stuck - feel free to tune
Erlang GC internals:
  http://www.erlang.org/doc/man/erts_alloc.html

> Weren't Channel design for that ? In our environment, we have (naively ?)
> considered the use of Channel to
> separate the message production from the consumption.
> Since we are targeting 10 000 peers doing both production and consumption,
> the fact of multiplying the number of
> connections by 2 is not negligible at all, considering scalability.
> Moreover, as I reported later on, we use SSL to authenticate the broker, and
> we are still unclear about memory leaks
> induce by SSL connections. Doubling the number of connections will not be
> negligible at all considering memory occupation either.
> In conclusion, we are not likely to implement our peers using 2 connections
> for the same broker.
> What would you recommend to us then ? And could you give us a better
> understanding on the use case of channels ?

Yes, channels were designed exactly for that. On the other hand, AMQP
has few pretty serious issues. For example when you open channel
you're free to publish a message. And broker can't 'refuse' accepting
a message. Channel.flow can be sent from the broker to the client
but *after* channel is opened. So there is a window in which
you just can publish (after channel.open before channel.flow). Sorry, there
just isn't other way of forbidding publishes than by stopping the whole
connection. I also don't like it, but it's the only way.


Cheers,
 Marek