[rabbitmq-discuss] exessive memory usage and blocked connection, even when no messages left

[Theo]

I rewrote my test code in Java, just to make sure there's nothing with
the Ruby driver that is causing the problem, here is the code:
https://gist.github.com/980238

I ran that with more or less the same result. After a few minutes at
20K publishes/deliveries/acks per second the cluster rotted and all
connections got blocked. What's more, this time I saw that if I turned
off my producers the web UI still reported messages being published
for several minutes -- no producers were running (I'm absolutely sure
the processes were terminated) but there were still messages being
published.

Theo

On May 18, 10:47 pm, Theo <icon... at gmail.com> wrote:
> Eventually (a couple of hours without any connections) the node
> returned to normal memory usage, so I launched a new batch of tests. I
> saw a sustained load of 20K published/delivered/acked for a few
> minutes, with the memory usage of each node increasing rapidly. After
> perhaps five minutes the cluster rotted completely. No messages
> published, nothing delivered, all connections are blocked.
>
> 20K messages over four nodes, that shouldn't be a problem? From my
> point of view it looks like RabbitMQ is leaking memory badly, but that
> couldn't be, could it?
>
> At this point there is no message on any queue, according to the web
> UI, but each node is still using over 3Gb RAM.
>
> Theo
>
> On May 18, 6:15 pm, Theo <icon... at gmail.com> wrote:
>
>
>
>
>
>
>
> > 2.4.1, should have been more specific. the OS is ubuntu 10.04, btw.
>
> > there's a lot of warnings in the log about memory usage, but it's more
> > or less the same thing over and over again:
>
> > =INFO REPORT==== 18-May-2011::14:37:20 ===
> > vm_memory_high_watermark set. Memory used:3327376104 allowed:
> > 3006570496
>
> > =INFO REPORT==== 18-May-2011::14:37:20 ===
> >     alarm_handler: {set,
> > {{vm_memory_high_watermark,rabbit at rfmmqstaging01},[]}}
>
> > then I found this, not sure if it's important (pasted it into a gist):
>
> >https://gist.github.com/978897
>
> > this is the version of erlang:
>
> > Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:8:8] [rq:8] [async-
> > threads:0] [hipe] [kernel-poll:false]
>
> > I've been watching the management UI since I posted my first message
> > (almost an hour ago) and the memory usage hasn't gone down. can a node
> > get bad? the other nodes look ok, but the one that's in the red in the
> > screenshot is still in the red. no connections to the server for an
> > hour.
>
> > T#
>
> > On May 18, 6:01 pm, Matthew Sackman <matt... at rabbitmq.com> wrote:> Hi Theo,
>
> > > On Wed, May 18, 2011 at 08:22:13AM -0700, Theo wrote:
> > > > More specifics on the setup: the cluster consists of 4 EC2 instances
> > > > with 8 CPUs and 7 Gb RAM each (I forget the exact instance name)
> > > > running RabbitMQ 2.4.
>
> > > Is that 2.4.0 or 2.4.1? If not 2.4.1, please upgrade. Which version of
> > > Erlang are you running? Is there anything of note other than connections
> > > and memory alarms in any of the logs?
>
> > > Matthew
> > > _______________________________________________
> > > rabbitmq-discuss mailing list
> > > rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss