[rabbitmq-discuss] Per-Connection Flow Control - RMQ 2.8.1

[DawgTool]

Hi Simon,

Here are some stats on a few runs, modifying the flow control limits.
Attached is some graphs I created to show the memory consumption.

Test 1-5 all on a 16core/96gb server

Publisher(s): 4
Exchange(s): 1
Exchange Type(s): Topic
Topic Wildcards: 1234.#; 2345.#; 3456.#; (etc)
Queue Binding(s): 15
Active Queue(s): 2
Queue Type(s): Durable, TTL: 90,000
Consumer(s): 0

Modified File(s): (variables: X, Y)
X: 4000; 8000; 16000; 32000
Y: 1000; 2000; 4000; 8000

src/credit_flow.erl
-define(DEFAULT_CREDIT, {"X", "Y"}).

src/rabbit_amqqueue.erl
-define(MORE_CONSUMER_CREDIT_AFTER, "Y").

src/rabbit_amqqueue_process.erl
-define(UNSENT_MESSAGE_LIMIT,          "X").

It appears that at some point the non-persistent data is dropped to disk 
(most cases were ~4GB total memory).
Locking is occurring before the 4GB mark, usually within a few minutes 
or seconds (depending on the flow control setting).

Please let me know if you would like to discuss.

Thanks

On 4/16/12 6:55 AM, Simon MacMullen wrote:
> I would have thought that in your scenario you would be more bound by 
> disc performance than anything else - aren't you having messages 
> paging off the disc and being expired?
>
> Just to be clear, do you have a message rate for your use case that 
> leads to *stable* memory use in 2.7.1 but which cannot be maintained 
> by 2.8.1? I was unable to find one with my attempts at replicating 
> your scenario.
>
> Cheers, Simon
>
> On 12/04/12 15:52, DawgTool wrote:
>> Hi Simon,
>>
>> Just a quick update. I have collected some stats, but mostly looking at
>> the rabbitmqctl list_channels. Anyway, I will get those graphed at some
>> point.
>>
>> I did however build the same setup (2.8.1/R15B) on a 16 core/96GB
>> machine and I hit the same limits as the 4 core/16GB.
>>
>> So right now it doesn't appear to be a 'horse-power' issue, possible its
>> a logic/functional issue. Let me know what you think.
>>
>> Thanks
>>
>>
>> On 3/28/12 8:42 AM, Simon MacMullen wrote:
>>> On 28/03/12 11:32, Simon MacMullen wrote:
>>>> I will try to reproduce based on the above, but anything else you can
>>>> tell me would be a great help.
>>>
>>> Hmm, I'm struggling with this.
>>>
>>> I can certainly get into a state with a combination of x-message-ttl,
>>> lots of messages expiring, and enough memory pressure to push much of
>>> the queue onto disc, such that the queue performance fluctuates quite
>>> a bit (as it switches between accepting new messages / paging messages
>>> out to disc / paging them back in / expiring them).
>>>
>>> But I can't get a message rate that works OK in this scenario with
>>> flow control disabled but blocks for long periods with it enabled. If
>>> the message rate is high enough for flow control to kick in, then it's
>>> also high enough for memory use to go up as messages wait to enter the
>>> queue if flow control is disabled. Eventually the memory alarm goes
>>> off instead.
>>>
>>> So can you provide more information about your use pattern?
>>>
>>> Having said that, if you just want to disable flow control and are
>>> prepared to build from source, you could apply the following patch:
>>>
>>> diff -r f0dafaca6cd1 src/rabbit_reader.erl
>>> --- a/src/rabbit_reader.erl Mon Mar 26 17:12:41 2012 +0100
>>> +++ b/src/rabbit_reader.erl Wed Mar 28 13:40:03 2012 +0100
>>> @@ -360,7 +360,7 @@
>>>
>>> control_throttle(State = #v1{connection_state = CS,
>>> conserve_memory = Mem}) ->
>>> - case {CS, Mem orelse credit_flow:blocked()} of
>>> + case {CS, Mem} of
>>> {running, true} -> State#v1{connection_state = blocking};
>>> {blocking, false} -> State#v1{connection_state = running};
>>> {blocked, false} -> ok = rabbit_heartbeat:resume_monitor(
>>>
>>> But I would like to understand more about what you're seeing.
>>>
>>> Cheers, Simon
>>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120423/761a3cc8/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: RabbitMQ2.8.1Custom.2012-04-23.jpg
Type: image/jpeg
Size: 697269 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120423/761a3cc8/attachment.jpg>