Thanks Jerry!<div>In which version it is fixed, can you please let me know.</div><div><br></div><div>Also, how do we get permission to see the bug details.</div><div><br></div><div>Regards,</div><div>Vinay<br><br><div class="gmail_quote">
On Sun, Jul 15, 2012 at 2:02 AM, Jerry Kuch <span dir="ltr"><<a href="mailto:jerryk@rbcon.com" target="_blank">jerryk@rbcon.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi, Vijay:<br><br>Initial segment of the discussion from the bug pasted here:<br><br><div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;margin-bottom:2em;font-weight:normal;line-height:normal;text-transform:none;font-size:small;white-space:normal;font-family:Verdana,sans-serif;word-spacing:0px">
<div style="padding-top:0.1em;padding-bottom:0.1em;padding-left:0.5em;background-color:rgb(224,224,224)"><span style="margin:0px 0.5em"><span><a href="mailto:steve@rabbitmq.com" title="Steve Powell <steve@rabbitmq.com>" style="color:rgb(102,51,102)" target="_blank"><span>Steve Powell</span></a><span> </span></span></span><span></span><span style="margin:0px 0.5em">2012-02-01 10:41:32 GMT</span></div>
<pre style="font-size:medium;font-family:monospace;white-space:pre-wrap;width:50em">See link, main part here:
public static void main(String[] argv) throws IOException {
ConnectionFactory f = new ConnectionFactory();
Connection c = f.newConnection();
for(int i = 0; i < 20000; ++i) {
Channel ch = c.createChannel();
ch.close();
}
System.gc();
while(true) {}
}
While program is running, observe the memory usage, it keeps on increasing
Get process id and do following
jmap -dump:format=b,file=yo.bin <pid>
jhat jhat -J-mx768m -stack false yo.bin
Once jhat brings up http server, go here
<a href="http://localhost:7000/showInstanceCounts/" style="color:rgb(102,51,102)" target="_blank">http://localhost:7000/showInstanceCounts/</a>
20001 instances of class com.rabbitmq.client.impl.AMQCommand
20001 instances of class com.rabbitmq.client.impl.CommandAssembler
20000 instances of class com.rabbitmq.client.ShutdownSignalException
20000 instances of class com.rabbitmq.client.impl.AMQImpl$Channel$Close
20000 instances of class com.rabbitmq.client.impl.ChannelN
20000 instances of class com.rabbitmq.client.impl.ConsumerDispatcher</pre></div><div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;margin-bottom:2em;font-weight:normal;line-height:normal;text-transform:none;font-size:small;white-space:normal;font-family:Verdana,sans-serif;word-spacing:0px">
<div style="padding-top:0.1em;padding-bottom:0.1em;padding-left:0.5em;background-color:rgb(224,224,224)"><span style="margin:0px 0.5em;float:right">[<a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#add_comment" style="color:rgb(102,51,102)" target="_blank">reply</a>]<span> </span><a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#" title="Collapse the comment." style="color:rgb(102,51,102);text-decoration:none" target="_blank">[-]</a></span><span style="margin:0px 0.5em;float:right"><a name="1388731f5a68cfa7_c1" href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c1" style="color:rgb(102,51,102)" target="_blank">Comment 1</a></span><span style="margin:0px 0.5em"><span><a href="mailto:steve@rabbitmq.com" title="Steve Powell <steve@rabbitmq.com>" style="color:rgb(102,51,102)" target="_blank"><span>Steve Powell</span></a><span> </span></span></span><span></span><span style="margin:0px 0.5em">2012-02-01 11:09:17 GMT</span></div>
<pre style="font-size:medium;font-family:monospace;white-space:pre-wrap;width:50em">(In reply to <a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c0" style="color:rgb(102,51,102)" target="_blank">comment #0</a>)
Ran sample test main under debug, and found the WorkPool in the
ConsumerWorkService contains all the AMQChannel objects!</pre></div><div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;margin-bottom:2em;font-weight:normal;line-height:normal;text-transform:none;font-size:small;white-space:normal;font-family:Verdana,sans-serif;word-spacing:0px">
<div style="padding-top:0.1em;padding-bottom:0.1em;padding-left:0.5em;background-color:rgb(224,224,224)"><span style="margin:0px 0.5em;float:right">[<a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#add_comment" style="color:rgb(102,51,102)" target="_blank">reply</a>]<span> </span><a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#" title="Collapse the comment." style="color:rgb(102,51,102);text-decoration:none" target="_blank">[-]</a></span><span style="margin:0px 0.5em;float:right"><a name="1388731f5a68cfa7_c2" href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c2" style="color:rgb(102,51,102)" target="_blank">Comment 2</a></span><span style="margin:0px 0.5em"><span><a href="mailto:steve@rabbitmq.com" title="Steve Powell <steve@rabbitmq.com>" style="color:rgb(102,51,102)" target="_blank"><span>Steve Powell</span></a><span> </span></span></span><span></span><span style="margin:0px 0.5em">2012-02-01 11:58:46 GMT</span></div>
<pre style="font-size:medium;font-family:monospace;white-space:pre-wrap;width:50em">Problem is we never issue stopWork(channel) on the ConsumerWorkService. This
means that the stuff in the WorkPool stays there.
Difficulty is that issuing this too early may mean that the last pieces of work
to be executed for a channel may be chopped. We may need to have a thread that
runs after channel.close() to tidy up when the last piece of work for that
channel is finished. This is only necessary if there is work in the pool for
that channel, otherwise we may simply stop work immediately.
Solution -- check WorkPool for work -- if there is none, there won't be any
more later. stopWork() and that's it.
If there is some work to be done, then schedule some more work which
automatically removes the channel from the work pool.</pre></div><div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;margin-bottom:2em;font-weight:normal;line-height:normal;text-transform:none;font-size:small;white-space:normal;font-family:Verdana,sans-serif;word-spacing:0px">
<div style="padding-top:0.1em;padding-bottom:0.1em;padding-left:0.5em;background-color:rgb(224,224,224)"><span style="margin:0px 0.5em;float:right">[<a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#add_comment" style="color:rgb(102,51,102)" target="_blank">reply</a>]<span> </span><a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#" title="Collapse the comment." style="color:rgb(102,51,102);text-decoration:none" target="_blank">[-]</a></span><span style="margin:0px 0.5em;float:right"><a name="1388731f5a68cfa7_c3" href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c3" style="color:rgb(102,51,102)" target="_blank">Comment 3</a></span><span style="margin:0px 0.5em"><span><a href="mailto:steve@rabbitmq.com" title="Steve Powell <steve@rabbitmq.com>" style="color:rgb(102,51,102)" target="_blank"><span>Steve Powell</span></a><span> </span></span></span><span></span><span style="margin:0px 0.5em">2012-02-01 14:59:50 GMT</span></div>
<pre style="font-size:medium;font-family:monospace;white-space:pre-wrap;width:50em">(In reply to <a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c2" style="color:rgb(102,51,102)" target="_blank">comment #2</a>)
<span style="color:rgb(101,55,156);white-space:pre-wrap">> Solution -- check WorkPool for work -- if there is none, there won't be any
> more later. stopWork() and that's it.</span>
<span style="color:rgb(101,55,156);white-space:pre-wrap">> If there is some work to be done, then schedule some more work which
> automatically removes the channel from the work pool.</span>
Actually, no, better is to hook into broadcastShutdownSignal() which calls
handleShutdownSignal even if there are no consumers. handleShutdownSignal()
can remove the work from the queue as it's last act.</pre></div><div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;margin-bottom:2em;font-weight:normal;line-height:normal;text-transform:none;font-size:small;white-space:normal;font-family:Verdana,sans-serif;word-spacing:0px">
<div style="padding-top:0.1em;padding-bottom:0.1em;padding-left:0.5em;background-color:rgb(224,224,224)"><span style="margin:0px 0.5em;float:right">[<a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#add_comment" style="color:rgb(102,51,102)" target="_blank">reply</a>]<span> </span><a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#" title="Collapse the comment." style="color:rgb(102,51,102);text-decoration:none" target="_blank">[-]</a></span><span style="margin:0px 0.5em;float:right"><a name="1388731f5a68cfa7_c4" href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c4" style="color:rgb(102,51,102)" target="_blank">Comment 4</a></span><span style="margin:0px 0.5em"><span><a href="mailto:steve@rabbitmq.com" title="Steve Powell <steve@rabbitmq.com>" style="color:rgb(102,51,102)" target="_blank"><span>Steve Powell</span></a><span> </span></span></span><span></span><span style="margin:0px 0.5em">2012-02-01 15:15:41 GMT</span></div>
<pre style="font-size:medium;font-family:monospace;white-space:pre-wrap;width:50em">Changeset <a href="http://hg.rabbitmq.com/rabbitmq-java-client/rev/0d949ba17115" style="color:rgb(102,51,102)" target="_blank">http://hg.rabbitmq.com/rabbitmq-java-client/rev/0d949ba17115</a>
Stop channel consumer work pool after channel.close()</pre></div><div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;margin-bottom:2em;font-weight:normal;line-height:normal;text-transform:none;font-size:small;white-space:normal;font-family:Verdana,sans-serif;word-spacing:0px">
<div style="padding-top:0.1em;padding-bottom:0.1em;padding-left:0.5em;background-color:rgb(224,224,224)"><span style="margin:0px 0.5em;float:right">[<a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#add_comment" style="color:rgb(102,51,102)" target="_blank">reply</a>]<span> </span><a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#" title="Collapse the comment." style="color:rgb(102,51,102);text-decoration:none" target="_blank">[-]</a></span><span style="margin:0px 0.5em;float:right"><a name="1388731f5a68cfa7_c5" href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c5" style="color:rgb(102,51,102)" target="_blank">Comment 5</a></span><span style="margin:0px 0.5em"><span><a href="mailto:steve@rabbitmq.com" title="Steve Powell <steve@rabbitmq.com>" style="color:rgb(102,51,102)" target="_blank"><span>Steve Powell</span></a><span> </span></span></span><span></span><span style="margin:0px 0.5em">2012-02-01 15:48:04 GMT</span></div>
<pre style="font-size:medium;font-family:monospace;white-space:pre-wrap;width:50em">Repeating tests for java client -- all appears well;
Reproducing debug run -- all appears cleared up;
Running jmap and jhat as shown -- we get lots of unresolved instance references
when running jhat (?), and shows:
10 instances of class
com.rabbitmq.client.impl.LongStringHelper$ByteArrayLongString
4 instances of class com.rabbitmq.client.impl.CommandAssembler$CAState
2 instances of class com.rabbitmq.client.DefaultSaslConfig
at top of instance counts list.
It looks fixed.</pre></div><div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">On Sat, Jul 14, 2012 at 1:00 PM, Vijay Lakshmi <span dir="ltr"><<a href="mailto:vijjivinay@gmail.com" target="_blank">vijjivinay@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Jerry,<div><br></div><div>Thanks for the response.</div><div>I can not browse the bug 24723 (it says access denied), can you please provide the details.<br>
<br>Regards,</div><div>Vinay</div><div><div><div><br><div class="gmail_quote">
On Sat, Jul 14, 2012 at 9:56 PM, Jerry Kuch <span dir="ltr"><<a href="mailto:jerryk@rbcon.com" target="_blank">jerryk@rbcon.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi, Vijay:<br><br>You may want to upgrade from 2.7.1. The 2.7.1 Java client had a known and now fixed memory leak that might account for what you're seeing:<br><br><a href="https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723" target="_blank">https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723</a><br>
<br>Best regards,<br>Jerry<br><br><div class="gmail_quote"><div><div>On Sat, Jul 14, 2012 at 7:35 AM, Vijay Lakshmi <span dir="ltr"><<a href="mailto:vijjivinay@gmail.com" target="_blank">vijjivinay@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>Hi,<div><br></div><div>We are using RabbitMQ 2.7.1 Java Client and Server 2.7.1.</div>
<div>While we were running load tests using our clients we observed that our memory has reached to 2GB.</div>
<div><br></div><div>Below is the snapshot of heap analysis.</div>
<div><br></div><div>We are closing the channels properly using channel.close().</div><div>Could someone suggest what could be the problem for this memory leak. Do the later versions of RabbitMQ solve this problem?</div><div>
<br></div><div><br></div><div><h1 style="margin-right:0in;margin-left:0in;font-size:24pt;font-family:'Times New Roman',serif;text-align:center" align="center"><span style="font-family:Times,serif">Instance Counts for All Classes (excluding platform)</span></h1>
<p class="MsoNormal" style="margin-top:0in;margin-bottom:0.0001pt;margin-right:0in;margin-left:0in;font-size:11pt;font-family:Calibri,sans-serif"><span><span style="font-size:13.5pt;font-family:Times,serif">311681<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77c6f06a0" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77c6f06a0" style="color:blue;text-decoration:underline" target="_blank">class com.rabbitmq.client.impl.AMQCommand</a> </span></span><span style="font-size:13.5pt;font-family:Times,serif"><br>
<span>311681<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77c6f3228" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77c6f3228" style="color:blue;text-decoration:underline" target="_blank">class com.rabbitmq.client.impl.CommandAssembler</a> </span><br>
<span>295730<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77d438380" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77d438380" style="color:blue;text-decoration:underline" target="_blank">class com.rabbitmq.client.impl.ChannelN</a> </span><br>
<span>295730<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77d43ff30" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77d43ff30" style="color:blue;text-decoration:underline" target="_blank">class com.rabbitmq.client.impl.ConsumerDispatcher</a> </span><br>
<span>295715<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77c60b280" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77c60b280" style="color:blue;text-decoration:underline" target="_blank">class com.rabbitmq.client.ShutdownSignalException</a> </span><br>
<span>295703<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77c75a540" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77c75a540" style="color:blue;text-decoration:underline" target="_blank">class com.rabbitmq.client.impl.AMQImpl$Channel$Close</a> </span><br>
<span>271315<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77d4d5590" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77d4d5590" style="color:blue;text-decoration:underline" target="_blank">class com.rabbitmq.client.QueueingConsumer</a> </span><br>
<span>60523<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77cea18a0" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77cea18a0" style="color:blue;text-decoration:underline" target="_blank">class org.json.simple.parser.Yytoken</a> </span><br>
<span>31600<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77c73fa58" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77c73fa58" style="color:blue;text-decoration:underline" target="_blank">class com.rabbitmq.client.AMQP$BasicProperties</a> </span><br>
<span>31210<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77d4d7358" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77d4d7358" style="color:blue;text-decoration:underline" target="_blank">class com.rabbitmq.client.QueueingConsumer$Delivery</a> </span><br>
<span>31209<a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77d4dc1d0" style="color:blue;text-decoration:underline" target="_blank"> instances</a> of <a href="http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77d4dc1d0" style="color:blue;text-decoration:underline" target="_blank">class com.rabbitmq.client.Envelope</a> </span></span></p>
<p class="MsoNormal" style="margin-top:0in;margin-bottom:0.0001pt;margin-right:0in;margin-left:0in;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:13.5pt;font-family:Times,serif"><span><br>
</span></span></p><p class="MsoNormal" style="margin-top:0in;margin-bottom:0.0001pt;margin-right:0in;margin-left:0in"><font face="Times, serif" size="4"><br></font></p></div>
<br></div></div>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br>
</div></div><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>