[rabbitmq-discuss] Per-Connection Flow Control - RMQ 2.8.1

Wed Apr 25 17:14:44 BST 2012

Hi Matthias,

On 4/25/12 11:50 AM, Matthias Radestock wrote:
> On 25/04/12 15:41, DawgTool wrote:
>> I haven't run these same tests on 2.7.1 yet, since its our production
>> machines (which are currently stable... enough).
>
> I see. So, just to be clear, we are now no longer looking at the 
> original problem you reported, namely that you thought 2.7.1 could 
> handle a considerably higher rate than 2.8.1?
>
Comparing 2.7.1 to 2.8.1 has become less a priority as getting 2.8.1 to 
behave the same as our current 2.7.1 system.
(and I say behave meaning throughput count more then anything else)
>> The spiking and the locking is not unusual? ;)
>> Maybe the large gaps between queue memory and actual memory is not
>> unusual also? ;)
>> Maybe the flat queue sizes for several seconds while hundreds of
>> thousands of records are added and purged? ;)
>> All joking aside.... =)
>
In case you thought different, this really is a good nature poke. So 
please don't take my joking personally. =)

> Running, as you were, with considerably higher flow control credits 
> than standard make all of the above worse. One of the reasons we 
> introduced flow control (and picked the standard settings we have) is 
> to reduce the magnitude of these.
Agreed, the reason I was raising the credits was to give my publishers 
more time to publish before they got locked out.  At the default level 
{200,50}, my publishing minimum rate of 4k/sec (normal rate is 90k/s) 
blocked almost instantly.
The flat queue concerns me more then the queue vs total, I expected that 
there would be records piled up in the broker/exchange.
>
>> When RabbitMQ is cleaning its index/journals and data files, the queues
>> are all locked.
>
> Paging to disk happens largely asynchronously. However, ordering and 
> other causality constraints may require certain operations to wait 
> until others have completed. There is not much we can do about that.
>
I assumed the same thing, 'purging'/paging would be something done 
async, but in all my tests with non-persistent messages, paging locked 
everything without fail, regardless of disk type (SAS vs RAM). I haven't 
gone through the code enough to know when is the case, but i would 
assume its the index/journal rebuilding and not the disk i/o.
>> The current flow control {200,50} hides a lot of this
>
> Yep. Quite intentionally.
>
>> On my test machine, all the non-persistent messages were dropped to disk
>> (causing a lock) around 4GB every time. On smaller systems, its usually
>> around 1.5 to 2GB. This happens regardless of any settings on rabbitmq
>> or mnesia (unless I missed something).
>
> With the default credit settings I can get publishers to pause for a 
> few seconds every now and then, but not longer. Are you seeing 
> something different?
You can easily get this to behave poorly by pushing volume (3-4k/s) to a 
topic exchange with a short TTL on the queues (90000ms)
I can send my json object if you want to see the configuration I am 
using. Summary: 1 exchange(topic) -> 14 queues (TTL:90000ms)
>
> Regards,
>
> Matthias.