[rabbitmq-discuss] Per-Connection Flow Control - RMQ 2.8.1

Fri Apr 27 15:42:51 BST 2012

Hi Marrhais,

On 4/25/12 12:36 PM, Matthias Radestock wrote:
> On 25/04/12 17:14, DawgTool wrote:
>> Comparing 2.7.1 to 2.8.1 has become less a priority as getting 2.8.1 to
>> behave the same as our current 2.7.1 system.
>> (and I say behave meaning throughput count more then anything else)
>
> *Sustained* throughput on 2.8.1 should not be significantly lower than 
> 2.7.1. Do you have evidence to the contrary? That is certainly 
> something we'd want to investigate.
>
> 2.7.1 can absorb longer spikes than 2.8.1; you can compensate for that 
> somewhat by increasing the tcp buffer sizes.
>
Agreed that 2.7.1 absorbs the spikes, which I think a message broker 
should be able to do. Accept the message, route when message is ready or 
queue is available.  Removing the flow control from 2.8.1 (or setting to 
some obscene amount) has similar results as 2.7.1, but having tested 
with the same configuration as production.
>> the reason I was raising the credits was to give my publishers
>> more time to publish before they got locked out. At the default level
>> {200,50}, my publishing minimum rate of 4k/sec (normal rate is 90k/s)
>> blocked almost instantly.
>
> What's the average message size? And how many queues does a single 
> message get routed to on average?
>
message: ~500 bytes (payload) + ~50 bytes 
(header:routekey,messageid,timestamp)

> 90k/s is unlikely to be a sustainable rate even with tiny messages. 
> 4k/sec really shouldn't be a problem, unless the messages are huge.
The current 2.7.1 uses several machines to consume from the publishers 
(pub) which feed to other machines holding the queues (que), consumers 
connect to another set of machines to consume (csm), this seems to work 
enough to keep the current rate of 90K/sec, but wouldn't take much to 
tip-over. All the machines are in the same named cluster, queues are 
hard coded to machines along with a single HA for each queue. This is 
not ideal, which is why we are looking for alternatives.
>
>> The flat queue concerns me more then the queue vs total, I expected that
>> there would be records piled up in the broker/exchange.
>
> One of the objectives of the credit based flow control is to prevent 
> messages from piling up anywhere other than the queue or tcp buffers.
>
>> You can easily get this to behave poorly by pushing volume (3-4k/s) to a
>> topic exchange with a short TTL on the queues (90000ms)
>
> Ah. TTLs. Handling those can be quite costly in some circumstances, 
> and indeed block the queue for a while. This shouldn't have changed 
> between 2.7.1 and 2.8.1 though.
We had enough room on the 2.7.1 cluster that I don't think we have ever 
hit a TTL, but that doesn't leave us room for a large failure in 
consumer processing, ie: loosing a whole rack of servers.

>
>
> Regards,
>
> Matthias.

Thanks