<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font face="Courier New, Courier, monospace">Hi Simon,</font><br>
<br>
<font face="Courier New, Courier, monospace">Here are some stats on
a few runs, modifying the flow control limits.</font><br>
<font face="Courier New, Courier, monospace">Attached is some graphs
I created to show the memory consumption.</font><br>
<br>
<font face="Courier New, Courier, monospace">Test 1-5 all on a
16core/96gb server</font><br>
<br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">Publisher(s): 4</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
Exchange(s): 1</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
Exchange Type(s): Topic</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">Topic Wildcards: 1234.#; 2345.#; 3456.#;
(etc)</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
Queue Binding(s): 15</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
Active Queue(s): 2</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
Queue Type(s): Durable, TTL: 90,000</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
Consumer(s): 0</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
Modified File(s): (variables: X, Y)</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
X: 4000; 8000; 16000; 32000</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
Y: 1000; 2000; 4000; 8000</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
src/credit_flow.erl</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
-define(DEFAULT_CREDIT, {"X", "Y"}).</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
src/rabbit_amqqueue.erl</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
-define(MORE_CONSUMER_CREDIT_AFTER, "Y").</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
src/rabbit_amqqueue_process.erl</font></font><br>
<font color="black" face="Courier New, Courier, monospace" size="2"><font
color="black" size="2">
-define(UNSENT_MESSAGE_LIMIT, "X").</font></font><br>
<br>
<font face="Courier New, Courier, monospace">It appears that at some
point the non-persistent data is dropped to disk (most cases were
~4GB total memory).</font><br>
<font face="Courier New, Courier, monospace">Locking is occurring
before the 4GB mark, usually within a few minutes or seconds
(depending on the flow control setting).</font><br>
<br>
<font face="Courier New, Courier, monospace">Please let me know if
you would like to discuss.</font><br>
<br>
<font face="Courier New, Courier, monospace">Thanks </font><br>
<br>
On 4/16/12 6:55 AM, Simon MacMullen wrote:
<blockquote cite="mid:4F8BFAB7.1070706@rabbitmq.com" type="cite">I
would have thought that in your scenario you would be more bound
by disc performance than anything else - aren't you having
messages paging off the disc and being expired?
<br>
<br>
Just to be clear, do you have a message rate for your use case
that leads to *stable* memory use in 2.7.1 but which cannot be
maintained by 2.8.1? I was unable to find one with my attempts at
replicating your scenario.
<br>
<br>
Cheers, Simon
<br>
<br>
On 12/04/12 15:52, DawgTool wrote:
<br>
<blockquote type="cite">Hi Simon,
<br>
<br>
Just a quick update. I have collected some stats, but mostly
looking at
<br>
the rabbitmqctl list_channels. Anyway, I will get those graphed
at some
<br>
point.
<br>
<br>
I did however build the same setup (2.8.1/R15B) on a 16
core/96GB
<br>
machine and I hit the same limits as the 4 core/16GB.
<br>
<br>
So right now it doesn't appear to be a 'horse-power' issue,
possible its
<br>
a logic/functional issue. Let me know what you think.
<br>
<br>
Thanks
<br>
<br>
<br>
On 3/28/12 8:42 AM, Simon MacMullen wrote:
<br>
<blockquote type="cite">On 28/03/12 11:32, Simon MacMullen
wrote:
<br>
<blockquote type="cite">I will try to reproduce based on the
above, but anything else you can
<br>
tell me would be a great help.
<br>
</blockquote>
<br>
Hmm, I'm struggling with this.
<br>
<br>
I can certainly get into a state with a combination of
x-message-ttl,
<br>
lots of messages expiring, and enough memory pressure to push
much of
<br>
the queue onto disc, such that the queue performance
fluctuates quite
<br>
a bit (as it switches between accepting new messages / paging
messages
<br>
out to disc / paging them back in / expiring them).
<br>
<br>
But I can't get a message rate that works OK in this scenario
with
<br>
flow control disabled but blocks for long periods with it
enabled. If
<br>
the message rate is high enough for flow control to kick in,
then it's
<br>
also high enough for memory use to go up as messages wait to
enter the
<br>
queue if flow control is disabled. Eventually the memory alarm
goes
<br>
off instead.
<br>
<br>
So can you provide more information about your use pattern?
<br>
<br>
Having said that, if you just want to disable flow control and
are
<br>
prepared to build from source, you could apply the following
patch:
<br>
<br>
diff -r f0dafaca6cd1 src/rabbit_reader.erl
<br>
--- a/src/rabbit_reader.erl Mon Mar 26 17:12:41 2012 +0100
<br>
+++ b/src/rabbit_reader.erl Wed Mar 28 13:40:03 2012 +0100
<br>
@@ -360,7 +360,7 @@
<br>
<br>
control_throttle(State = #v1{connection_state = CS,
<br>
conserve_memory = Mem}) ->
<br>
- case {CS, Mem orelse credit_flow:blocked()} of
<br>
+ case {CS, Mem} of
<br>
{running, true} -> State#v1{connection_state = blocking};
<br>
{blocking, false} -> State#v1{connection_state = running};
<br>
{blocked, false} -> ok = rabbit_heartbeat:resume_monitor(
<br>
<br>
But I would like to understand more about what you're seeing.
<br>
<br>
Cheers, Simon
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
<br>
</blockquote>
<br>
</body>
</html>