[rabbitmq-discuss] Problem stopping and starting rabbitmq (branch bug21673)

[Matthew Sackman]

Hi Scott,

On Mon, Feb 15, 2010 at 12:24:00PM -0800, scott w wrote:
> On Mon, Feb 15, 2010 at 12:03 PM, Matthew Sackman <matthew at lshift.net>wrote:
> > They should do the same thing. This one is very odd - it looks like tcp
> > listeners are being started as the Erlang VM kernel is being shutdown.
> > I've never seen this before, but I'd be slightly surprised if it's
> > unique to 21673. Can you reproduce this at all on the default branch?
> 
> Unfortunately no. To be honest, everything works fine on small loads. We are
> using that branch for very large loads where the number of tasks don't fit
> into memory. So while the problem itself may not be specific to this branch,
> I haven't seen this problem in the default branch because the default can't
> handle the loads we're generating.

Right. We have another suspicion, that you may be stopping rabbit with
rabbitmqctl stop_app and then using something in /etc/init.d to start
Rabbit. stop_app only stops the rabbit application running within the
Erlang VM. It doesn't stop the erlang VM itself. Hence the difference
between rabbitmqctl stop and rabbitmqctl stop_app. Could that be what's
going on?

> > Also, which version of Erlang are you running? 12B3 is the minimum for
> > that branch, and 13B03 is recommended.
> >
> 
> 12B-4 is what we're building from source. What benefits would we get from
> upgrading to 13B03?

Thousands of bug fixes, and a much better GC.

> Btw, are there any special rabbitmq/erlang commands or tools we can run to
> verify that rabbimq is healthy beyond running rabbitmqctl and looking at
> logs? I find the logs rather cryptic sometimes.

Generally, provided something like rabbitmqctl list_queues is
responsive, all is well. If that starts taking several seconds to
respond then Rabbit's under heavy load, but that doesn't mean anything's
going wrong, just that it's got a lot to do.

We've always written Rabbit very defensively - if it starts to go wrong,
it'll explode hard, quickly. So in general, if it's up and running and
doing something, then it's usually quite happy.

Matthew