[rabbitmq-discuss] RabbitMQ Client 2.7.1 Memory Leak?

Sat Jul 14 21:32:23 BST 2012

Hi, Vijay:

Initial segment of the discussion from the bug pasted here:

Steve Powell <steve at rabbitmq.com> 2012-02-01 10:41:32 GMT

See link, main part here:

public static void main(String[] argv) throws IOException {
   ConnectionFactory f = new ConnectionFactory();
   Connection c = f.newConnection();
   for(int i = 0; i < 20000; ++i) {
       Channel ch = c.createChannel();
       ch.close();
   }
   System.gc();
   while(true) {}
}

While program is running, observe the memory usage, it keeps on increasing

Get process id and do following
jmap -dump:format=b,file=yo.bin  <pid>
jhat jhat -J-mx768m -stack false yo.bin
Once jhat brings up http server, go
herehttp://localhost:7000/showInstanceCounts/

20001 instances of class com.rabbitmq.client.impl.AMQCommand
20001 instances of class com.rabbitmq.client.impl.CommandAssembler
20000 instances of class com.rabbitmq.client.ShutdownSignalException
20000 instances of class com.rabbitmq.client.impl.AMQImpl$Channel$Close
20000 instances of class com.rabbitmq.client.impl.ChannelN
20000 instances of class com.rabbitmq.client.impl.ConsumerDispatcher

[reply <https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#add_comment>]
[-] <https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#>Comment
1<https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c1>Steve
Powell <steve at rabbitmq.com> 2012-02-01 11:09:17 GMT

(In reply to comment #0
<https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c0>)

Ran sample test main under debug, and found the WorkPool in the
ConsumerWorkService contains all the AMQChannel objects!

[reply <https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#add_comment>]
[-] <https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#>Comment
2<https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c2>Steve
Powell <steve at rabbitmq.com> 2012-02-01 11:58:46 GMT

Problem is we never issue stopWork(channel) on the ConsumerWorkService.  This
means that the stuff in the WorkPool stays there.

Difficulty is that issuing this too early may mean that the last pieces of work
to be executed for a channel may be chopped.  We may need to have a thread that
runs after channel.close() to tidy up when the last piece of work for that
channel is finished.  This is only necessary if there is work in the pool for
that channel, otherwise we may simply stop work immediately.

Solution -- check WorkPool for work -- if there is none, there won't be any
more later.  stopWork() and that's it.

If there is some work to be done, then schedule some more work which
automatically removes the channel from the work pool.

[reply <https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#add_comment>]
[-] <https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#>Comment
3<https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c3>Steve
Powell <steve at rabbitmq.com> 2012-02-01 14:59:50 GMT

(In reply to comment #2
<https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c2>)> Solution --
check WorkPool for work -- if there is none, there won't be any
> more later.  stopWork() and that's it.
> If there is some work to be done, then schedule some more work which
> automatically removes the channel from the work pool.

Actually, no, better is to hook into broadcastShutdownSignal() which calls
handleShutdownSignal even if there are no consumers.  handleShutdownSignal()
can remove the work from the queue as it's last act.

[reply <https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#add_comment>]
[-] <https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#>Comment
4<https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c4>Steve
Powell <steve at rabbitmq.com> 2012-02-01 15:15:41 GMT

Changeset http://hg.rabbitmq.com/rabbitmq-java-client/rev/0d949ba17115

Stop channel consumer work pool after channel.close()

[reply <https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#add_comment>]
[-] <https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#>Comment
5<https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723#c5>Steve
Powell <steve at rabbitmq.com> 2012-02-01 15:48:04 GMT

Repeating tests for java client -- all appears well;
Reproducing debug run -- all appears cleared up;
Running jmap and jhat as shown -- we get lots of unresolved instance references
when running jhat (?), and shows:

10 instances of class
com.rabbitmq.client.impl.LongStringHelper$ByteArrayLongString
4 instances of class com.rabbitmq.client.impl.CommandAssembler$CAState
2 instances of class com.rabbitmq.client.DefaultSaslConfig

at top of instance counts list.

It looks fixed.

On Sat, Jul 14, 2012 at 1:00 PM, Vijay Lakshmi <vijjivinay at gmail.com> wrote:

> Hi Jerry,
>
> Thanks for the response.
> I can not browse the bug 24723 (it says access denied), can you please
> provide the details.
>
> Regards,
> Vinay
>
> On Sat, Jul 14, 2012 at 9:56 PM, Jerry Kuch <jerryk at rbcon.com> wrote:
>
>> Hi, Vijay:
>>
>> You may want to upgrade from 2.7.1.  The 2.7.1 Java client had a known
>> and now fixed memory leak that might account for what you're seeing:
>>
>> https://bugzilla.rabbitmq.com/show_bug.cgi?id=24723
>>
>> Best regards,
>> Jerry
>>
>> On Sat, Jul 14, 2012 at 7:35 AM, Vijay Lakshmi <vijjivinay at gmail.com>wrote:
>>
>>> Hi,
>>>
>>> We are using RabbitMQ 2.7.1 Java Client and Server 2.7.1.
>>> While we were running load tests using our clients we observed that our
>>> memory has reached to 2GB.
>>>
>>> Below is the snapshot of heap analysis.
>>>
>>> We are closing the channels properly using channel.close().
>>> Could someone suggest what could be the problem for this memory leak. Do
>>> the later versions of RabbitMQ solve this problem?
>>>
>>>
>>> Instance Counts for All Classes (excluding platform)
>>>
>>> 311681 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77c6f06a0>
>>>  of class com.rabbitmq.client.impl.AMQCommand<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77c6f06a0>
>>>
>>> 311681 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77c6f3228>
>>>  of class com.rabbitmq.client.impl.CommandAssembler<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77c6f3228>
>>>
>>> 295730 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77d438380>
>>>  of class com.rabbitmq.client.impl.ChannelN<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77d438380>
>>>
>>> 295730 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77d43ff30>
>>>  of class com.rabbitmq.client.impl.ConsumerDispatcher<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77d43ff30>
>>>
>>> 295715 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77c60b280>
>>>  of class com.rabbitmq.client.ShutdownSignalException<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77c60b280>
>>>
>>> 295703 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77c75a540>
>>>  of class com.rabbitmq.client.impl.AMQImpl$Channel$Close<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77c75a540>
>>>
>>> 271315 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77d4d5590>
>>>  of class com.rabbitmq.client.QueueingConsumer<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77d4d5590>
>>>
>>> 60523 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77cea18a0>
>>>  of class org.json.simple.parser.Yytoken<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77cea18a0>
>>>
>>> 31600 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77c73fa58>
>>>  of class com.rabbitmq.client.AMQP$BasicProperties<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77c73fa58>
>>>
>>> 31210 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77d4d7358>
>>>  of class com.rabbitmq.client.QueueingConsumer$Delivery<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77d4d7358>
>>>
>>> 31209 instances<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2finstances%2f0x77d4dc1d0>
>>>  of class com.rabbitmq.client.Envelope<http://redir.aspx?C=dba9a335a30e4feb822ceaa30de7f654&URL=http%3a%2f%2flocalhost%3a7000%2fclass%2f0x77d4dc1d0>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>
>>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120714/67074802/attachment.htm>